Matlab mldivide returns NaN when there are multiple solutions

Your matrix A is singular. The MATLAB doc states that in these cases, mldivide is unreliable ("... When working with ill-conditioned matrices, an unreliable solution can result ...") and suggests to use lsqminnorm( ) or pinv( ) instead (see "Tips"). E.g.,

>> A = [ -1 0  0;
    1 1 -1;
    0 -1 1 ];
>> B = [-1; 0; 1];
>> A\B
Warning: Matrix is singular to working precision. 
 
ans =
   NaN
   NaN
   NaN
>> A*ans-B
ans =
   NaN
   NaN
   NaN
>> lsqminnorm(A,B)
ans =
    1.0000
   -0.5000
    0.5000
>> A*ans-B
ans =
   1.0e-15 *
    0.4441
   -0.3331
   -0.1110
>> pinv(A)*B
ans =
    1.0000
   -0.5000
    0.5000
>> A*ans-B
ans =
   1.0e-15 *
    0.2220
   -0.4441
    0.2220