All of the content has been moved And reformatted. And lightly edited to clean up a few of the long-standing problems. Moved to https://slott56.github.io/. Nothing new will be posted here. All the new stuff will go to https://slott56.github.io/. Fix Your Bookmarks Why are you still reading? Click the link and go to https://slott56.github.io/ and fix your bookmarks ..read more

Check out DZone's Coding zone: https://dzone.com/coding. Hover over the "Coding" drop-down menu. Notice anything lacking? I'll give you a hint: Python. They have "Frameworks", "Java", "Javascript", "Languages" and "Tools". The "Languages" seems to be general programming, and the posts include Java, JavaScript and numerous other languages.  It leaves me with questions about the basis for the "Zones". It can't be popularity, since Java has slipped behind Python. Maybe there's some other criteria; I wonder what they could be? Sponsorships? Or the historical "Javalobby" web site?  I ..read more

Apple Books Pivot to Python A Guide for professionals and skilled beginners https://books.apple.com/us/book/pivot-to-python/id1586977675  I've recently updated this to fix some cosmetic problems with title pages, the table of contents and stuff like that. The content hasn't changed. Yet. It's still an introduction to Python for folks who already know how to program, they want to pivot to programming in Python. Quickly. But wait, there's more.  Unlearning SQL When your only tool is a hammer, every problem looks like a nail https://books.apple.com/us/book/unlearning-sql/id6443164060 Ma ..read more

Some background: FAERIE DUST™, Obstinate Idiocy, Obstinate Idiocy, Expanded, and even Permutations, Combinations and Frustrations. I want to set up algorithm design as the diametric opposite of Obstinate Stupidity. To do that, let's look at Obstinate Stupidity. The theme?  We did something wrong, and we don't want to fix it. I emphasize this because it takes many forms. Another common variant is "We can't afford to continue the way we are, but we can't afford the time to fix it, either." Sometimes, "Management wants this fixed, but we don't have any budget." You know how it is. The curren ..read more

Real Question (somewhat abbreviated): "My algorithm performs badly, do I need asyncio?" Short answer: No. Long answer: Sigh. No. Do you need a slap upside the head? Here's how it plays out: Q: "We figured that if we 'parallelize' it, then we can apply multiple cores, and it will run 4x as fast." Me: "What kind of I/O are you doing?" Q: "None, really. It's compute-intensive." Me: "Async is for I/O. A function can be computing while other functions are waiting for I/O to complete." Q: "Right. We can have lots of them, so they each get a core." Me: "Listen, please. A function can be computing. Th ..read more

This isn't about details of pySpark. This is about the philosophy of testing when working with a large, complex framework, like pySpark, pandas, numpy, or whatever.  BLUF Use data subsets.  Write unit tests for the functions that process the data. Don't test pyspark itself. Test the code you write. Some History I've worked with folks -- data scientists specifically -- without a deep background in software engineering. When we said their model-building applications needed a test case, they supplied the test case they used to validate the model. Essentially, their test script ..read more

See https://towardsdatascience.com/building-generator-pipelines-in-python-8931535792ff  I'm delighted by this article.  I was shown only the first, horrible, example. I think the idea was to push back on the idea of complex generators. I fumed. Then I read the entire article. Now I'm fuming at someone who posted the first example -- apparently having failed to read the rest of the post. This idea of building a stack of iterators is very, very good. The example (using simple operations) can be misleading. A follow-on example doing something like file parsing might be helpful. But, if ..read more

I've been trying to help some folks who have a "search" algorithm that's slow.  They know it's slow -- that's pretty obvious. They're -- unfortunately -- sure that asyncio will help. That's not an obvious conclusion. It involves no useful research. Indeed, that's a kind of magical thinking. Which leads me to consider the process of over-engineering. The Problem Over-engineering is essentially a technique for burning brain-calories on planning to build something instead of building something. The distinction is "planning" vs. "doing." Lots of folks subscribe to Methodology Magic Thinking ..read more

I started with the Apple Books platform because it's an easy default for me.  Pivot to Python A Guide for professionals and skilled beginners https://books.apple.com/us/book/pivot-to-python/id1586977675  I've recently updated this to fix some cosmetic problems with title pages, the table of contents and stuff like that. The content hasn't changed. Yet. It's still an introduction to Python for folks who already know how to program, they want to pivot to programming in Python. Quickly. But wait, there's more.  Unlearning SQL When your only tool is a hammer, every problem looks lik ..read more

In the long run, I think there's a small book here. See 8 reasons Python will rule the enterprise — and 8 reasons it won’t | InfoWorld. The conclusion, "Teams need to migrate slowly into the future, and adopting more Python is a way to do that," seems to be sensible. Some of the cautionary tales along the way, however, don't make as much sense. TL;DR. There are no reasons to avoid Python. Indeed, the 8 points suggest that Python is perhaps a smart decision.  I want to focus on the negatives part of this because some of them are wrong. I think there's a "technology hegemony" viewpoint wher ..read more