A journey to find the origin of a broken angular app

Recently I got alerted that a form on our website was misbehaving and not working. We got reports that the submission was "ignored" and the parameters ended in the location bar. It started to look like some javascript explosion.

DynaTrace was able to provide some information that enabled us to orient our investigations.

The collected errors

We started searching for the symptoms (parameters in url) in the Web Requests section. As we are only sampling a fraction of our users, not all web requests stored are linked to client collected data. It is manual work to make the link as searching/filtering capacities of DynaTrace is quite bad in that situation.

We were able to collect some client errors that were somewhat related to the misbehavior.

It is interesting to note that DynaTrace, like the other error collection tools, save the stack as-is, meaning that the message are localized.

load error

TypeError: L'objet ne gère pas la propriété ou la méthode << load >>
   at Anonymous function (Unknown script code:1:79)
   at Global code (Unknown script code:1:2)
   at Anonymous function (https://www.googletagmanager.com/gtm.js?id=GTM-XXXX&l=gtmDataObject:79:380)
   at bi (https://www.googletagmanager.com/gtm.js?id=GTM-XXXXXX&l=gtmDataObject:81:188)
   at zf (https://www.googletagmanager.com/gtm.js?id=GTM-XXXXXX&l=gtmDataObject:49:38)
   at Anonymous function (https://www.googletagmanager.com/gtm.js?id=GTM-XXXX&l=gtmDataObject:120:792)
   at Fk (https://www.googletagmanager.com/gtm.js?id=GTM-XXXXXX&l=gtmDataObject:116:192)
   at Tk (https://www.googletagmanager.com/gtm.js?id=GTM-XXXXXX&l=gtmDataObject:120:1565)
   at gg (https://www.googletagmanager.com/gtm.js?id=GTM-XXXXXX&l=gtmDataObject:121:201)
   at tg (https://www.googletagmanager.com/gtm.js?id=GTM-XXXXXX&l=gtmDataObject:60:128)

Somebody's using load on a undefined object.

Access Denied

Error: Accès refusé.

   at Anonymous function (https://www.example.com/libs.js?v=16.14.14:10:24895)
   at Global code (https://www.example.com/libs.js?v=16.14.14:10:24864)

That one is tricky, see below

Quota Exceeded

Error: Mémoire insuffisante pour cette opération.

   at Anonymous function (https://www.example.com/libs.js?v=16.14.14:10:24895)
   at Global code (https://www.example.com/mosaic/libs.js?v=16.14.14:10:24864)

The french and Dutch version of the English "Quota exceeded" error becomes "Insufficient memory for this operation" which is clearly better.

This is related to localStorage or sessionStorage being limited see wikipedia web storage.

Angular error

Error: [$injector:modulerr] Failed to instantiate module userIsLogged due to:
Error: [$injector:modulerr] Failed to instantiate module RequestUtils due to:
Error: [$injector:nomod] Module 'RequestUtils' is not available! You either misspelled the module name or forgot to load it. If registering a module ensure that you specify the dependencies as the second argument.
http://errors.angularjs.org/1.2.28/$injector/nomod?p0=RequestUtils
   at Anonymous function (https://www.example.com/libs.js?v=16.14.14:2:9891)
   at e (https://www.example.com/libs.js?v=16.14.14:2:9564)
   at Anonymous function (https://www.example.com/libs.js?v=16.14.14:2:9811)
   at Anonymous function (https://www.example.com/libs.js?v=16.14.14:2:23748)
   at w (https://www.example.com/libs.js?v=16.14.14:2:1663)
   at r (https://www.example.com/libs.js?v=16.14.14:2:23683)

Hmmm... A part of our angular application handling the form was complaining on injected module being undefined. But why? The injected module being undefined, the whole form logic was dead and reduced to its HTML tags with their default behaviors.

The reason of the form misbehaving was now clear: the form tag did not have a default method nor action and the result on clicking on submit was that the current page was requested again with the form parameter in the query string.

I now had to be able to reproduce the problem locally to investigate further in order to fix it.

Reproducing the failure

In order to reproduce, we had to isolate, if possible, browsers, pages, actions to try find the curse of non-trivial events leading the form failure.

The reason of the form misbehaving was now clear: the form tag did not have default method nor action and the result on submitting it was that the current page was requested again with the form parameter in the query string.

The first error (load) was excluded from the research as it occurred on a separated inline script and therefore not crashing the rest of the script of the page.

The "access denied" failure was mostly related to IE (Trident and Edge).

The "quota exceeded" affect nearly all browsers.

The last 2 errors are related to Web Storage, implemented in sessionStorage and localStorage.

I knew about issue in several browser (e.g. Safari in private mode) that just breaks the storage. We were using something like the following simplified example:

var tools = (function() {
  var storage = window.localStorage;

  try {
    storage.setItem('test', 'test');
    storage.removeItem('test');
    return storage;
  }
  catch(e) {
    return {
      getItem: function(key) {},
      setItem: function() {},
      // you get the picture
    }
  }
})();

This covers most of the issues:

1. when the web storage is disable (then the value is null);
2. when setItem throws an error (e.g. Safari private mode);
3. when setItem throws the quota error (most browsers)

But ONE was not: IE sometimes throwing "Access denied" when you "mention" window.localStorage. The line var storage = window.localStorage; throwed with the consequence of stopping the execution of the file.

Just needed to find when.

Digging a little more in the IE/dark side I discovered that it is possible to disable the Web Storage completely by policy. For the interested ones, you can do it with the following command line in windows (see windows protected mode):

icacls %userprofile%\Appdata\LocalLow /t /setintegritylevel (OI)(CI)M

Finally we were able to reproduce a failure similar to the reported one.

The correction was to enhance the wrapper aroung Web Storage in order to catch all failure cases and fallback from localStorage to sessionStorage to dummyStorage (storage on window).

/**
 * The aim of this is to expose safe localStorage and sessionStorage
 * The cases are:
 * - "Access Denied" on mention of window.localStorage (IE in secure mode)
 * - null returned when requesting window.localStorage or window.sessionStorage (DOM storage disabled)
 * - error on usage of .setItem (e.g. Safari sometimes or Quota exceeded)
 */
/**
 * yes, this dummy storage does not expose the complete Storage API but it
 * should suite most of our use-cases
 * @returns {Storage}
 */
var dummyStorage = function(w) {
  var localWindow = w || window;
  var keyName = '__dummyStorage__';
  localWindow[keyName] = localWindow[keyName] || {};
  return {
    getItem: function(id) {
      return localWindow[keyName][id] || null;
    },
    setItem: function(id, value) {
      localWindow[keyName][id] = value;
    },
    removeItem: function(id) {
      delete localWindow[keyName][id];
    }
  };
};
/**
 * @returns {Storage}
 */
var safeSessionStorage = function(w) {
  var localWindow = w || window;
  var now = Date.now();
  try {
    localWindow.sessionStorage.setItem('test-' + now, '1234');
    localWindow.sessionStorage.removeItem('test-' + now);
    return localWindow.sessionStorage;
  }
  catch (e) {
    return dummyStorage(localWindow);
  }
};
/**
 * @returns {Storage}
 */
var safeLocalStorage = function(w) {
  var localWindow = w || window;
  var now = Date.now();
  try {
    localWindow.localStorage.setItem('test-' + now, '1234');
    localWindow.localStorage.removeItem('test-' + now);
    return localWindow.localStorage;
  }
  catch (e) {
    // this will catch any error
    return safeSessionStorage(localWindow);
  }
};

Wrap up

The outcomes of this investigation

1. always make sure that you are satisfied with default HTML behaviors of you code in case your javascript fails (even make it work without javascript!)
2. Be always very cautious when using browser provided features -- very easy to say after the facts ;-)
3. There are a lot more issues in your scripts in production than you'd expect, have some simple tool to collect it, if only to have a metric.
4. Third party while not always being the guilty part, introduce a lot of noise in the bug hunting party (load error was a third party script not being very cautious)

References

• wikipedia web storage
• windows protected mode
• Always catch LocalStorage...
• Using web storage MDN