TLDR: If you don't want to understand the under-the-hood explanation, here's what you've been waiting for: you can use threading if your program is...
For further actions, you may consider blocking this person and/or reporting abuse
Probably a silly question, but, does your multi-threaded code work properly on Python for Windows? I once tried to port a parallel-tasking tool I'd written to run on Linux - where it ran quite well - but when I tried to execute it, on Windows, it blew up because of Windows' lack of a proper
vfork()implementation/analogue. I never had the time to revisit the issue to sort out how to get the thread-spawning speed I needed. Just wondering if you'd run into such issues and how you worked around them.vfork is a child creating syscall, which means is used to create new processes, not new threads.
Was it called manually somehow? Because multi-threading and multi-processing code in Python is cross platform.
I was just using the standard python mp libraries. Worked on Linux; blew the hell up on Windows. This woulda been at least 30 months ago. So, maybe something's changed in Python?
Well, it might have, 30 months is a long time in software development :-)
And yet, when you're in software development (or whatever the hell it is I do), you so rarely have time to actually revisit code. If you're lucky/cursed enough to have a memory like mine, you'll remember a problem you previously ran into and it will cause you to drop a drive-by question at someone who you assess to have a higher probability of knowing the answer (or, at least, where to find it).
Thanks for replying. Assuming I get time to revisit, I'll have to see if my previously broken code now works. =)
Old unmaintaned code is usually a source of headaches, because you might have multiple libraries that have been updated since, in addition to the language itself.
But you might get lucky :)
Fortunately, it's wholly unused code: the project in question was me attempting to merge (and then deprecate) two, platform-specific solutions with a cross-platform solution. Since it blew up when I tried to run it on Windows (and the platform-specific ones were still serviceable) and it was more a "for me" project than customer-demanded, it got shoved to the bottom of the priority pile. In truth, it was one of those "it's dead at work, lemme dork around and see if I can spiff some things" kind of project.
Maybe as we hire more dev-staff, I'll actually be allocated time to revisit some of my earlier "get it the hell out the door" and some of my "this woulda been a good idea to implement" tools.
Thanks for the article!
Bonus mentions: ThreadPoolExecutor and ProcessPoolExecutor - they allow you to easily create a pool of threads (the first) or processes (the latter) and to submit work to them.
You can also do something similar with multiprocessing.Pool if you use processes.
Memory copy is popular issue we need aware when use multiprocessing, it's problem of 'fork' system call when spawn new process, not Python. A rule is always calling multiprocessing.Process before you load your huge data