DEV Community

Discussion on: I'm an Expert in Memory Management & Segfaults, Ask Me Anything!

rishabh900 profile image
all_yours

I invoked valgrind by specifying it as an option in the launch file itself and the same way i invoked gdb.

<node name="gazebo" pkg="gazebo_ros"  type="$(arg script_type)" respawn="$(arg respawn_gazebo)" output="$(arg output)" launch-prefix="valgrind"

I am very stressed with this problem but i don't want to give up.
What do you suggest for this problem?

Thread Thread
codemouse92 profile image
Jason C. McDonald Ask Me Anything • Edited

You know, I'd be really curious to know what would happen if you ran the launch file itself through Valgrind! If you look at the output from a moment ago, there's quite a lot that is occuring outside of Valgrind (all the lines not preceded with ==nnnnn== (where nnnnn is some number). The segfault at the end appears to be occuring outside of that context as well. That leads me to believe the segfault might actually be within the launch file.

Thread Thread
rishabh900 profile image
all_yours

I just ran it through valgrind

Thread Thread
Sloan, the sloth mascot
Comment deleted
Sloan, the sloth mascot
Comment deleted
Sloan, the sloth mascot
Comment deleted
Sloan, the sloth mascot
Comment deleted
Sloan, the sloth mascot
Comment deleted
Sloan, the sloth mascot
Comment deleted
Sloan, the sloth mascot
Comment deleted
Sloan, the sloth mascot
Comment deleted
codemouse92 profile image
Jason C. McDonald Ask Me Anything

Yikes. Could you delete that comment chain and put it in a Gist or bpaste.net or some such? It'll be easier to read.

In any case, that confirmed my suspicion; the launcher is the problem. it's not memory pure at all.

Thread Thread
Sloan, the sloth mascot
Comment deleted
Sloan, the sloth mascot
Comment deleted
rishabh900 profile image
all_yours

After ending the process manually I further got the output

^C[robo_state_publisher-4] killing on exit
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done
==15557== Invalid read of size 4
==15557==    at 0x41964F: PyObject_Free (in /usr/bin/python2.7)
==15557==    by 0x4D07FA: ??? (in /usr/bin/python2.7)
==15557==    by 0x4AA262: ??? (in /usr/bin/python2.7)
==15557==    by 0x4E0C11: ??? (in /usr/bin/python2.7)
==15557==    by 0x4FC2C9: _PyModule_Clear (in /usr/bin/python2.7)
==15557==    by 0x4FBADC: PyImport_Cleanup (in /usr/bin/python2.7)
==15557==    by 0x4F8D83: Py_Finalize (in /usr/bin/python2.7)
==15557==    by 0x4936F1: Py_Main (in /usr/bin/python2.7)
==15557==    by 0x507782F: (below main) (libc-start.c:291)
==15557==  Address 0x62f2020 is 2,592 bytes inside an unallocated block of size 2,768 in arena "client"
==15557== 
==15557== Invalid read of size 4
==15557==    at 0x41964F: PyObject_Free (in /usr/bin/python2.7)
==15557==    by 0x4AA262: ??? (in /usr/bin/python2.7)
==15557==    by 0x4E0C11: ??? (in /usr/bin/python2.7)
==15557==    by 0x4FC2C9: _PyModule_Clear (in /usr/bin/python2.7)
==15557==    by 0x4FBADC: PyImport_Cleanup (in /usr/bin/python2.7)
==15557==    by 0x4F8D83: Py_Finalize (in /usr/bin/python2.7)
==15557==    by 0x4936F1: Py_Main (in /usr/bin/python2.7)
==15557==    by 0x507782F: (below main) (libc-start.c:291)
==15557==  Address 0x7f37020 is 128 bytes inside a block of size 552 free'd
==15557==    at 0x4C2EDEB: free (vg_replace_malloc.c:530)
==15557==    by 0x50C4362: fclose@@GLIBC_2.2.5 (iofclose.c:84)
==15557==    by 0x43CEFB: ??? (in /usr/bin/python2.7)
==15557==    by 0x4A63FD: PyObject_Call (in /usr/bin/python2.7)
==15557==    by 0x5385A5: _PyObject_CallMethod_SizeT (in /usr/bin/python2.7)
==15557==    by 0x53F4CE: ??? (in /usr/bin/python2.7)
==15557==    by 0x4AEF42: PyObject_CallFunctionObjArgs (in /usr/bin/python2.7)
==15557==    by 0x4BF668: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4BA915: PyEval_EvalCodeEx (in /usr/bin/python2.7)
==15557==    by 0x4C2C3B: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4BA915: PyEval_EvalCodeEx (in /usr/bin/python2.7)
==15557==    by 0x4C24E9: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==  Block was alloc'd at
==15557==    at 0x4C2DB8F: malloc (vg_replace_malloc.c:299)
==15557==    by 0x50C4CDC: __fopen_internal (iofopen.c:69)
==15557==    by 0x53D247: ??? (in /usr/bin/python2.7)
==15557==    by 0x4AB6FA: ??? (in /usr/bin/python2.7)
==15557==    by 0x53CDBE: ??? (in /usr/bin/python2.7)
==15557==    by 0x4BD1D9: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4BA915: PyEval_EvalCodeEx (in /usr/bin/python2.7)
==15557==    by 0x4C2C3B: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4BA915: PyEval_EvalCodeEx (in /usr/bin/python2.7)
==15557==    by 0x4C24E9: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4BA915: PyEval_EvalCodeEx (in /usr/bin/python2.7)
==15557==    by 0x4C24E9: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557== 
==15557== Invalid read of size 4
==15557==    at 0x41964F: PyObject_Free (in /usr/bin/python2.7)
==15557==    by 0x4FC2C9: _PyModule_Clear (in /usr/bin/python2.7)
==15557==    by 0x4FBADC: PyImport_Cleanup (in /usr/bin/python2.7)
==15557==    by 0x4F8D83: Py_Finalize (in /usr/bin/python2.7)
==15557==    by 0x4936F1: Py_Main (in /usr/bin/python2.7)
==15557==    by 0x507782F: (below main) (libc-start.c:291)
==15557==  Address 0x7eea020 is 0 bytes inside a block of size 8 free'd
==15557==    at 0x4C2EDEB: free (vg_replace_malloc.c:530)
==15557==    by 0x49B1E4: ??? (in /usr/bin/python2.7)
==15557==    by 0x4D878E: ??? (in /usr/bin/python2.7)
==15557==    by 0x4BD778: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4BA915: PyEval_EvalCodeEx (in /usr/bin/python2.7)
==15557==    by 0x4D6218: ??? (in /usr/bin/python2.7)
==15557==    by 0x4EEC7D: ??? (in /usr/bin/python2.7)
==15557==    by 0x4A63FD: PyObject_Call (in /usr/bin/python2.7)
==15557==    by 0x4C6C2F: PyEval_CallObjectWithKeywords (in /usr/bin/python2.7)
==15557==    by 0x6EB480C: ??? (in /usr/lib/python2.7/lib-dynload/pyexpat.x86_64-linux-gnu.so)
==15557==    by 0x6EBCF3D: ??? (in /usr/lib/python2.7/lib-dynload/pyexpat.x86_64-linux-gnu.so)
==15557==    by 0x710D68F: ??? (in /lib/x86_64-linux-gnu/libexpat.so.1.6.0)
==15557==  Block was alloc'd at
==15557==    at 0x4C2DB8F: malloc (vg_replace_malloc.c:299)
==15557==    by 0x493F0E: PyList_New (in /usr/bin/python2.7)
==15557==    by 0x510D4D: ??? (in /usr/bin/python2.7)
==15557==    by 0x4BD1D9: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4BA915: PyEval_EvalCodeEx (in /usr/bin/python2.7)
==15557==    by 0x4D6218: ??? (in /usr/bin/python2.7)
==15557==    by 0x4EEC7D: ??? (in /usr/bin/python2.7)
==15557==    by 0x4A63FD: PyObject_Call (in /usr/bin/python2.7)
==15557==    by 0x4C6C2F: PyEval_CallObjectWithKeywords (in /usr/bin/python2.7)
==15557==    by 0x6EB480C: ??? (in /usr/lib/python2.7/lib-dynload/pyexpat.x86_64-linux-gnu.so)
==15557==    by 0x6EBCF3D: ??? (in /usr/lib/python2.7/lib-dynload/pyexpat.x86_64-linux-gnu.so)
==15557==    by 0x710D68F: ??? (in /lib/x86_64-linux-gnu/libexpat.so.1.6.0)
==15557== 
==15557== Invalid read of size 4
==15557==    at 0x41964F: PyObject_Free (in /usr/bin/python2.7)
==15557==    by 0x4D0C94: ??? (in /usr/bin/python2.7)
==15557==    by 0x4D086B: ??? (in /usr/bin/python2.7)
==15557==    by 0x4FC2C9: _PyModule_Clear (in /usr/bin/python2.7)
==15557==    by 0x4FBADC: PyImport_Cleanup (in /usr/bin/python2.7)
==15557==    by 0x4F8D83: Py_Finalize (in /usr/bin/python2.7)
==15557==    by 0x4936F1: Py_Main (in /usr/bin/python2.7)
==15557==    by 0x507782F: (below main) (libc-start.c:291)
==15557==  Address 0x611c020 is 48,736 bytes inside a block of size 49,152 free'd
==15557==    at 0x4C2EDEB: free (vg_replace_malloc.c:530)
==15557==    by 0x4AA3C4: ??? (in /usr/bin/python2.7)
==15557==    by 0x495BCA: PyDict_SetItem (in /usr/bin/python2.7)
==15557==    by 0x4FC278: _PyModule_Clear (in /usr/bin/python2.7)
==15557==    by 0x4FBADC: PyImport_Cleanup (in /usr/bin/python2.7)
==15557==    by 0x4F8D83: Py_Finalize (in /usr/bin/python2.7)
==15557==    by 0x4936F1: Py_Main (in /usr/bin/python2.7)
==15557==    by 0x507782F: (below main) (libc-start.c:291)
==15557==  Block was alloc'd at
==15557==    at 0x4C2FB55: calloc (vg_replace_malloc.c:711)
==15557==    by 0x498D2C: ??? (in /usr/bin/python2.7)
==15557==    by 0x4A252E: PyDict_Merge (in /usr/bin/python2.7)
==15557==    by 0x512275: ??? (in /usr/bin/python2.7)
==15557==    by 0x4BD1D9: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4C210E: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4BA915: PyEval_EvalCodeEx (in /usr/bin/python2.7)
==15557==    by 0x4C2C3B: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4C210E: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4C210E: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4C210E: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557==    by 0x4C210E: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==15557== 
==15557== Invalid read of size 4
==15557==    at 0x41964F: PyObject_Free (in /usr/bin/python2.7)
==15557==    by 0x4D0B76: ??? (in /usr/bin/python2.7)
==15557==    by 0x4D086B: ??? (in /usr/bin/python2.7)
==15557==    by 0x495BCA: PyDict_SetItem (in /usr/bin/python2.7)
==15557==    by 0x4FC278: _PyModule_Clear (in /usr/bin/python2.7)
==15557==    by 0x4FBBBD: PyImport_Cleanup (in /usr/bin/python2.7)
==15557==    by 0x4F8D83: Py_Finalize (in /usr/bin/python2.7)
==15557==    by 0x4936F1: Py_Main (in /usr/bin/python2.7)
==15557==    by 0x507782F: (below main) (libc-start.c:291)
==15557==  Address 0x6083020 is 3,200 bytes inside a block of size 3,218 free'd
==15557==    at 0x4C2EDEB: free (vg_replace_malloc.c:530)
==15557==    by 0x4D0A85: ??? (in /usr/bin/python2.7)
==15557==    by 0x4D086B: ??? (in /usr/bin/python2.7)
==15557==    by 0x495BCA: PyDict_SetItem (in /usr/bin/python2.7)
==15557==    by 0x4FC278: _PyModule_Clear (in /usr/bin/python2.7)
==15557==    by 0x4FBBBD: PyImport_Cleanup (in /usr/bin/python2.7)
==15557==    by 0x4F8D83: Py_Finalize (in /usr/bin/python2.7)
==15557==    by 0x4936F1: Py_Main (in /usr/bin/python2.7)
==15557==    by 0x507782F: (below main) (libc-start.c:291)
==15557==  Block was alloc'd at
==15557==    at 0x4C2DB8F: malloc (vg_replace_malloc.c:299)
==15557==    by 0x4A0021: PyString_FromStringAndSize (in /usr/bin/python2.7)
==15557==    by 0x4B3F50: ??? (in /usr/bin/python2.7)
==15557==    by 0x4B425C: ??? (in /usr/bin/python2.7)
==15557==    by 0x4B414F: ??? (in /usr/bin/python2.7)
==15557==    by 0x4B4272: ??? (in /usr/bin/python2.7)
==15557==    by 0x4B3E65: PyMarshal_ReadObjectFromString (in /usr/bin/python2.7)
==15557==    by 0x4B3DE5: PyMarshal_ReadLastObjectFromFile (in /usr/bin/python2.7)
==15557==    by 0x4B3D2D: ??? (in /usr/bin/python2.7)
==15557==    by 0x4B390B: ??? (in /usr/bin/python2.7)
==15557==    by 0x4A4C20: ??? (in /usr/bin/python2.7)
==15557==    by 0x4A42B2: PyImport_ImportModuleLevel (in /usr/bin/python2.7)
==15557== 
==15557== Conditional jump or move depends on uninitialised value(s)
==15557==    at 0x419658: PyObject_Free (in /usr/bin/python2.7)
==15557==    by 0x4D0C94: ??? (in /usr/bin/python2.7)
==15557==    by 0x4D086B: ??? (in /usr/bin/python2.7)
==15557==    by 0x4AA262: ??? (in /usr/bin/python2.7)
==15557==    by 0x4E0C11: ??? (in /usr/bin/python2.7)
==15557==    by 0x4AA0E3: ??? (in /usr/bin/python2.7)
==15557==    by 0x4E0BFB: ??? (in /usr/bin/python2.7)
==15557==    by 0x4AA262: ??? (in /usr/bin/python2.7)
==15557==    by 0x4E0C11: ??? (in /usr/bin/python2.7)
==15557==    by 0x4FC2C9: _PyModule_Clear (in /usr/bin/python2.7)
==15557==    by 0x4FBBBD: PyImport_Cleanup (in /usr/bin/python2.7)
==15557==    by 0x4F8D83: Py_Finalize (in /usr/bin/python2.7)
==15557== 
==15557== Invalid read of size 4
==15557==    at 0x502477: PyGrammar_RemoveAccelerators (in /usr/bin/python2.7)
==15557==    by 0x4F8DF3: Py_Finalize (in /usr/bin/python2.7)
==15557==    by 0x4936F1: Py_Main (in /usr/bin/python2.7)
==15557==    by 0x507782F: (below main) (libc-start.c:291)
==15557==  Address 0x615a020 is 304 bytes inside a block of size 617 free'd
==15557==    at 0x4C2EDEB: free (vg_replace_malloc.c:530)
==15557==    by 0x4D07FA: ??? (in /usr/bin/python2.7)
==15557==    by 0x4FC2C9: _PyModule_Clear (in /usr/bin/python2.7)
==15557==    by 0x4FBADC: PyImport_Cleanup (in /usr/bin/python2.7)
==15557==    by 0x4F8D83: Py_Finalize (in /usr/bin/python2.7)
==15557==    by 0x4936F1: Py_Main (in /usr/bin/python2.7)
==15557==    by 0x507782F: (below main) (libc-start.c:291)
==15557==  Block was alloc'd at
==15557==    at 0x4C2DB8F: malloc (vg_replace_malloc.c:299)
==15557==    by 0x4A0021: PyString_FromStringAndSize (in /usr/bin/python2.7)
==15557==    by 0x4B3F50: ??? (in /usr/bin/python2.7)
==15557==    by 0x4B407D: ??? (in /usr/bin/python2.7)
==15557==    by 0x4B4272: ??? (in /usr/bin/python2.7)
==15557==    by 0x4B414F: ??? (in /usr/bin/python2.7)
==15557==    by 0x4B4272: ??? (in /usr/bin/python2.7)
==15557==    by 0x4B3E65: PyMarshal_ReadObjectFromString (in /usr/bin/python2.7)
==15557==    by 0x4B3DE5: PyMarshal_ReadLastObjectFromFile (in /usr/bin/python2.7)
==15557==    by 0x4B3D2D: ??? (in /usr/bin/python2.7)
==15557==    by 0x4B390B: ??? (in /usr/bin/python2.7)
==15557==    by 0x4A4C20: ??? (in /usr/bin/python2.7)
==15557== 
==15557== 
==15557== HEAP SUMMARY:
==15557==     in use at exit: 3,393,686 bytes in 5,832 blocks
==15557==   total heap usage: 278,397 allocs, 272,565 frees, 371,732,938 bytes allocated
==15557== 
==15557== LEAK SUMMARY:
==15557==    definitely lost: 0 bytes in 0 blocks
==15557==    indirectly lost: 0 bytes in 0 blocks
==15557==      possibly lost: 55,704 bytes in 96 blocks
==15557==    still reachable: 3,337,982 bytes in 5,736 blocks
==15557==         suppressed: 0 bytes in 0 blocks
==15557== Rerun with --leak-check=full to see details of leaked memory
==15557== 
==15557== For counts of detected and suppressed errors, rerun with: -v
==15557== Use --track-origins=yes to see where uninitialised values come from
==15557== ERROR SUMMARY: 9917 errors from 128 contexts (suppressed: 0 from 0)

So this was the whole output I got , sorry for uploading this in parts(character limitation).

I hope this gives something useful to track down the issue.

Thread Thread
rishabh900 profile image
all_yours

I apologise for making such a long comment chain.
I have now made a gist of running the launch file through valgrind in

gist.github.com/rishabh900/41fd6df...

And the above comment is the output after i terminated the process manually.
So what do you think of now?

Thread Thread
codemouse92 profile image
Jason C. McDonald Ask Me Anything • Edited

Did you write the launcher script, or is that third-party? It's clearly written in Python, and the issue is definitely there. I just can't narrow in on the specific issue, because the memory issues are being thrown by the interpreter (e.g. at 0x41964F: PyObject_Free (in /usr/bin/python2.7)). That indicates that something odd has been done within the Python code, but I won't be able to diagnose this further without really fully understanding the launcher's source code, and I'm afraid I don't have time to learn it.

If this is third-party code, open an issue against the launcher project, and include the above output of Valgrind.