Evan Klein | Blog

Grad School Check-in: 40% Done

Tue, 06 May 2025 00:00:00 GMT

Grad School Check-in: ~40% Done

Yesterday I submitted my last assignments for my graduate school classes, and I realized I haven’t written much here about the experience of getting my Masters (Data Science @ IU). I only have one week before my summer semester starts, so I thought I’d pump out a quick piece about some of the interesting things I’ve gotten to do so far.

Big picture

I find school to be very fulfilling, and while my program is very challenging (especially when balancing with a full-time job), it’s frankly nothing compared to my undergraduate years. Mostly I notice that I have a lot less bullshit-time; time spent on my phone, or sitting on my couch. Giving up real time with family, hobbies, etc. hasn’t really hit me, though I do occasionally have to give up a round of golf. I chose a good time in my life to do this.

Courses

Two semesters in, two classes each. So far I’ve taken:

D590: Introduction to Python Programming: Frankly a pretty easy ride for me given my software development experience (Python was my first language, which I first started learning back in 2012). I’m actually glad I took this, mostly for the exposure it gave me to libraries like pandas and matplotlib. I haven’t written much Python code in my career, but that will likely change, and it was good to start off easy and freshen up those skills.
S519: A Gentle Introduction to Statistics in R: I enjoyed this course a lot, but “gentle” is a very subjective term. I struggled even with basic probability problems, and it was quite the adjustment carving out my week so that I routinely had 8+ hours to complete my assignments. The skills in this class I are essential, and I really enjoyed the opportunity to learn R.
S580: Introduction to Regression Models and Nonparametrics: The sequel course to S519. More challenging, more dense, but you get to do some very fun things in this course. I feel like what I took away in S519 was very loose and abstract - concepts really came together here once you get to put things together and build models. Nonparametric testing and modeling methods are also very cool, and felt a little more “computer science”-y to me than some of the more distributional or probabilistic modeling that got touched on at the end of S519. This isn’t a required course in my program, I took it as an elective, but I’m glad that I did because I feel some of the statistical weaknesses that held me back my first semester really has been addressed due to the concepts here. I feel like I’ll be able to take my data science skills to the next level now that I feel comfortable with ideas like logistic regression and generalized additive models. Shoutout to Dr. Luen - easily my favorite professor so far.
I535: Management, Access, and Use of Big Data: This course was all about breadth, not depth, in dealing with all the different pieces of data management. It touched on a little bit of everything - cloud, virtualization, lifecycles, pipelines, governance, security, and more. I’ve enjoyed this class the least, so far. The lectures were okay, but oftentimes they consisted of random YouTube videos or Ted talks that only half-related to the module content. I liked the assignments, and the final project had very loose requirements (giving me some time to build some cool stuff), but largely I wouldn’t take this class again if I was going to do this program from the start. I just felt like I knew a little too much about most modules to get anything new.

Cool stuff I’ve gotten to build

K-means cluster algorithm: machine learning for breast cancer detection

For D590, our final project was to build a machine learning algorithm to detect breast cancer in subjects using a dataset of cellular samples. K-means clustering involves projecting attributes of an entity into an n-dimensional space and finding similar samples that live in that same “space” using euclidean distance (or other distance-based heuristics).

The project involved performing exploratory data analysis, processing and cleaning data, and then training the algorithm before testing its efficacy. My resulting implementation was able to successfully predict whether cell samples were benign or malignant with an overall 96.0% accuracy.

This was a great project for learning about a basic machine learning algorithm and how it might be used in practice.

Watchdog: financial and economic pipelines and dashboards with Python and Blazor

Like I said, the final project for I535 was loose enough that students could pretty much build whatever they want, given that they could relate it back to the course materials in an interesting way. I chose to build a series of pipelines for scraping economic, financial, and trade data from multiple sources like the Bureau of Labor Statistics, the U.S. Census Bureau, and the SEC Edgar Database. I actually think this could be an interesting product if it had some more work put into it. Some key features:

Give users the ability to study U.S. trade patterns across commodities (imports and exports)
Give users the ability to examine mutual and echange-traded fund holdings, including stocks, bonds, and other securities.
Give users the ability to examine various economic indicators and CPI averages.

Statistical Analysis of Economic Convergence in R

This was some of the most fun I’ve had so far - validating claims about the theory of economic convergence using real data, in R, for S580. I’ve really come around on R. There’s some syntactical funkiness for sure (wtf is with the 1-indexed arrays), but when it comes to doing real statistical work this is ultimately where I’m comfortable. The project involved studying OECD data around GDP for most countries to determine whether poor countries are more capable of explosive growth than larger countries. The report I authored determined that, given the data I had access to, the theory of economic convergence could not be confirmed with statistical methods like linear regression.

Up next

I’m projected to graduate next Spring, at my current pace, which is admittedly very fast. Most students in my program take one class a semester, but I’d rather not be in school for 2+ more years.

This summer I’m taking a course on data visualization, and a series of mini-courses that span all sorts of different topics, which I’m sure I’ll come back and write all about. I’m expecting the next two semesters to be very hard, but I’m looking forward to learning some new skills and wrapping this thing up soon. Thanks for reading!

Effective, Enforced Documentation Control with LLMs

Sat, 01 Feb 2025 00:00:00 GMT

Effective, Enforced Documentation Control with LLMs

I would not describe myself as a particularly talented software engineer. Of the production code I’ve written, I would describe it as 15% pretty bad, 5% pretty good, and 80% passable. This is more a reflection of the profession than I think it is demonstrative of my ability. Writing code is hard. Reading code is even harder. I think this is why despite a successful career thus far I still often feel like I’m just not any good at this. But I still try, and I’m always looking for little ways to leg up. LLMs have been useful. What I really find useful is good documentation. More than anything, when I’m trying to understand a system or process, what I want is a meticulously defined markdown file or PDF that explains it to me effectively. We don’t always have this luxury, but think that there’s a way in this age that we could, using LLMs and something called enforced documentation control.

I would like to start this conversation by entertaining two contradictory ideas about documentation:

Documentation is good

Finding good documentation invokes a practically euphoric chemical reaction in my brain. Complex technical topics and systems, explained clearly in readable, human language is one of the best tools we have for understanding such topics and systems. Documentation about a very specific system that I happen to be working on? Look at those thousands of lines of code I didn’t have to read, because some developer was nice enough to reason it out on paper.

More than that, documentation serves as a bridge between different types of people. It connects non-technical people and technical people. It connects experienced engineers with junior engineers. It bridges gaps in knowledge in a way that’s much more permanent. An engineer could get hit by a bus, taking all of his ideas and knowledge about his work with him. The documentation he wrote stays behind, where it can hopefully nurture and assist others.

Idea #1: Every codebase should have a rich literature of documentation that accompanies it.

Documentation is bad

Lots of developers hate writing documentation, and for good reason. Writing documentation is usually a little bit boring, and tedious. It takes a lot of time, time that could be better spent writing code. There’s also the fact that a lot of documentation is bullshit. And I don’t mean bullshit colloquially, I mean it in a technical sense. If you’ve never heard the technical definition of bullshit, it’s expressed in Harry Frankfurt’s essay On Bullshit. He defines bullshit as “speech without regards to the truth”. This is different from a lie, where the speaker knows the truth of a matter and violates it anyways. I feel this effect on documentation is rarely due to an engineer’s purposeful, malevolent effort. The bullshitification of software documentation happens much more subtly, and is probably more easily attributed to priorities, or just straight up laziness. Take this example:


    /// <summary>
    /// Sends the same email to all addresses in a distribution list, logs any relevant errors. Fails gracefully on a per-email basis.
    /// </summary>
    /// <param name="distributionList"></param>
    /// <param name="subject"></param>
    /// <param name="message"></param
    private async Task SendEmailsToDistributionList(IEnumerable<string> distributionList,
        string subject, string message)
    {
        for (string address in distributionList)
        {
            try
            {
                bool status = await EmailService.SendEmailAsyncWithRetry(address, subject, message, retry = 5);
                if (!status)
                {
                    Log.Error($"Error sending email to {address}.");
                }
            }
            catch (EmailException ex)
            {
                Log.Error(ex, $"Exception when sending email to {address}.")
            }
        }
    }

This code takes a subject and message and passes it to the injected IEmailService so that it can send the same message to all the email addresses in a distribution list. Since our function is allowed to fail gracefully (and silently), it might actually make more sense to move our logging into the email service, so that other parts of our app that use the IEmailService don’t have to implement their own logging for failures. This saves on redundant code. So now our function looks like this:


    /// <summary>
    /// Sends the same email to all addresses in a distribution list, logs any relevant errors. Fails gracefully.
    /// </summary>
    /// <param name="distributionList"></param>
    /// <param name="subject"></param>
    /// <param name="message"></param
    private async Task SendEmailsToDistributionList(IEnumerable<string> distributionList,
        string subject, string message)
    {
        for (string address in distributionList)
        {
            await EmailService.SendEmailAsyncWithRetry(address, subject, message, retry = 5, failQuietely = true);
        }
    }

I think this is a nice change (as an example at least). Do you see the problem we’ve just created though? The header documentation for the function is now wrong. What’s worse - it’s only wrong in an implicit sense. Nothing about the functionality has changed - all the same logic from before the change is included here, just abstracted away into a different part of the codebase. This function summary is not lying - but it’s misleading. These are the types of little things that make documentation just a little bit less trustworthy - a little bit more bullshit than they ought to be.

As authors of software, we are only a single code change from rendering our documentation bullshit. We could make the extra effort to check our documentation anytime a code change is made, but we don’t always do that. Changes like this can easily slip through a code review. Code is material; it exists in the physical world. When it breaks, so do our lives, because suddenly something is broken right now and we have to fix it. Customers are affected. Money is lost. Documentation can break too, but there’s never as great an incentive or urgency, because it’s largely immaterial. We don’t always notice when it’s incorrect. When it breaks, the consequences often never come at all, or they blindside us way down the line.

Additionally - broken documentation is actually worse than no documentation, because it is wrong information that presents itself as the truth. Broken documentation is a false positive.

If you’re an engineer, there’s a tempting answer: screw the documentation! I’ll just write my code in a “self-documenting” style. I don’t disagree with that idea. You should try and make your code as simple and explanatory as possible. But this idea doesn’t really scale - you can’t always self-document an entire system or application.

Idea #2: There are a lot of very good reasons to not document your code.

Don’t give up

The problem with documentation, as I’ve stated, is that it is immaterial - its purpose is to serve as a helpful abstraction; a human-readable representation of a complex system. If the system changes, the onus is on the documentation to change and stay current. If the documentation changes however, there’s no enforcement for the software to also change - it’s happy to keep tumbling along, behaving in the exact way it always has.

But what if we flipped things on its head? What if we made our documentation as material as our code? Maybe it would look a little something like this:

You can probably see where I’m going with this, but just to spoil it: I think LLMs have made technical documentation very relevant as part of the build process, and maybe even the code review process. What if you never merged unless you knew your documentation and your code were in sync? Previously, the only way to do this was to spend as much time verifying documentation as you did writing features and fixing bugs. But if you had a reliable model running this for you, then that’s a lot of manual work you don’t have to do.

Making Documentation Real

If we want to care about writing good quality code that works, and writing effective documentation that stands on its own, we need to square this circle: we need to make our documentation as real as our code.

Here’s a sketch of how I think this could work:

In your repo, create a Documentation directory, and fill it with documentation for your project. Fill it with best practices, API references, microservice definitions, common patterns, security documentation, architectural diagrams, flowcharts. Put a whole document in here explaining why your team uses spaces or tabs. Write a document where you tell the story of that one time the intern broke the deployment pipeline because he changed the new release’s version number to v🍾.🥴.🤮.
Incorporate a Documentation Control system, powered by your LLM of choice. The half-assed recreation above shows it as a blocker on a PR (probably created via a GitHub action), but it could a CLI tool, or simply incorporated into your build pipeline of choice.
Run your DC control system ad-hoc, or as part of your CI. Track conflicts between documentation and code the exact same way that you might track conflicts between code in different branches. If there are conflicts, force the author to do one of two things:
1. Update the code to be in-line with the documentation
2. Update the documentation to be in-line with the code (being careful to think about other places that the DC system might throw out conflicts if this route is taken)

I’m choosing to leave a lot up to the imagination here, because this is a silly idea that I don’t have the time to really implement and interface with. Off-the-cuff, there’s a few issues I can think of:

LLMs themselves are kinda bullshit
Security issues; preventing your LLM from ingesting unique, sensitive material about the inner workings of your system, and regurgitating it for a bad actor.
The idea of a “documentation conflict” has the potential to be far less binary than something like a typical merge conflict in source control. How do you address the grays in-between?
Culture. I can imagine some variation of this conversation would probably play out in Teams and Slack worldwide if we had this:

Dave: “Hey Marty, I just opened a PR. Can you review?”

Marty: “Sure thing! Code looks good. breathes in sharply through teeth Ooh, looks like you’ve got a bunch of documentation conflicts.”

Dave: “Yeah don’t worry about that, it’s just a bunch of bullshit.”

Marty: “Okay lmao” *squash*

I don’t want to hedge too much, because I do think that these problems can be addressed. LLMs are getting better every day. A lead that cares about documentation conflicts as much as he does clean code might be able to implement this idea successfully across his org.

Documentation-Driven Development: Made Possible

In addition to being a useful maintenance and review tool, I think it might even possible to take this idea one step further. What if your project was built documentation-first? Instead of writing code from stories derived from requirements, you write documentation from those requirements. Then you build features, branching frequently, and only merging to main if:

Your code passes code review, conducted by other engineers.
Your code passes documentation review, conducted by your model.

Seems strange, and I can’t really begin to unpack all the possible problems at this stage. I haven’t even worked within a TDD or BDD type environment long enough to know how I feel about that particular style of development. But I think if you had a reliable model, something you know with reasonable certainty you can trust to effectively manage your documentation control, then maybe it could work. At this stage, I think the former use case as a repository maintenance tool is probably more realistic and useful, but I started this idea with something like documentation-driven development, and it feels wrong to abandon outright just because I can’t parse it fully.

I’m also not the first person to come up with something like this. Like I mentioned, BDD is a thing (along with ATDD). Perhaps a documentation control system (I also like to think of it as an enforcer) could augment these methodologies, or maybe it could be spun off into its own thing like I’ve described above. At this point, I’m just in ideas land.

Conclusion

To bring it back home, strict documentation has its goods and bads. I think this approach - enforced documentation control, has the ability to eliminate a lot of the bad aspects, under the right circumstances. It has the ability to force more careful consideration of code and system design, both in the micro and macroscopic senses, and it’s a way to effectively bridge technical knowledge between technical people (and non-technical people) in a way that does not get stale or begin to smell. We audit systems and codebases all the way time - why not docs too?

Year in Review: 2024

Tue, 24 Dec 2024 00:00:00 GMT

A Year in Review: 2024

This is my first year with the new site, and I thought I’d continue stealing ideas from other blogs I like by doing a “Year in Review”. No guidelines, except for writing about what I want to write. Its good to keep records.

Blog Posts

This year I wrote 7 posts, including this one. Honestly, it’s not a lot, but I’m fine with it. I made this site so that I’d have somewhere to post anything and everything from random thoughts and essays to fleshed out technical guides, but that doesn’t mean I should be putting out things for the sake of putting them out. The internet is full of pointless shit that doesn’t or shouldn’t be said, so there’s no need to add to the noise.

Career

I moved jobs this year; my first real move in my career. Quitting my first job was one of the hardest things I’ve ever done. After I formally put in my notice, I began to do the rounds to colleagues to let them know myself that I’d soon be gone. I turned back to my office, whom I shared with my friend and mentor Tom, and saw him standing side-by-side with my boss. They both had their arms crossed, and gazed at me with dissappointed, but understanding looks. That stung. It can be hard to say goodbye to people you care about and who have been so good to you. I had to fight very hard not to feel shame and guilt over ‘abandoning’ those who had given me every opportunity to succeed in the first stage of my career. Saying goodbye is never easy. But it was the right thing to do. I care more about my work now. I’m more driven. I have a new team around me that I get to learn new things from. I have new opportunities. I got what I needed.

One of the harder things was going from hybrid/1-day in the office to hybrid/3-days. If I’m brutally honest with myself, I was kidding myself when I ever said that “only 1 day in the office” was good for me. Going to 3 days in the office has been a challenge, and I feel like the first few years of my career being fully-remote maybe wired me to reject the office as a place I want to be. I have found that I have a really hard time now just sitting still, surrounded by other people in noisy shared spaces. Even when I want to be productive, it can be a struggle, sometimes even physically. My first month, I suffered from frequent headaches, I can only imagine from the white fluorescent lights and brring noise machines. Physiologically, my body was telling me that this place was bad. Go home. Go back to where it’s comfortable. But like I said, working from home is not the best for me. Despite the new challenges, I do enjoy going into the office, and I do find myself getting more work done there than at home. I wouldn’t trade in those remaining two WFH days for anything - they’re good days for interspersing chores and recharing your social battery. A hybrid approach is definitely the way to go. But I’ll need a long time to fully adjust to my new situation.

Back to School

This Fall, I started graduate school at Indiana University. I’m pursuing my master’s degree in data science. There’s a lot of reasons I wanted to do this. I tossed around ideas of an MS in cybersecurity or an MBA first. I spoke to a lot of people about this decision, including personal friends and family and colleagues, and the only real takeaway I got was to do the same thing I did when I chose Purdue for undergraduate: pick something I actually cared about and go learn how to do it. Data science seemed like a good fit, once that realization became clear.

Realistically, I’m pretty proud of this decision. It’s not something I ever thought about doing until one day I decided I wanted to. More than anything, this is a hard thing that I chose to pursue. Life is full of doing things, some hard and some easy, but most of them are things you kinda have to do, even if it might be presented as a choice. Choosing to do something challenging, when there’s not an immediate need to, comes with the rewarding, but sometimes bittersweet feeling of knowing you did it to yourself. So far, I have had no regrets with my choice.

My program has been challenging, but also really enjoyable. I took a Python data science class, as well as a graduate-level statistics course that taught me some of the R programming language. What I’ve learned is that juggling school and work feels much different than just doing school full-time. I’ve definitely lost some of my math chops, and my time management needs work and consistency. Some units I submitted assignments two weeks early, and some units I submitted work two hours before the deadline. That kind of dispersion is not great. I’m really looking forward to next semester, where I’ll be tackling more statistics (non-parametrics) and some big data concepts.

Projects

I managed to build a couple of cool things this year.

The Blog

This year I built this blog! It’s an Astro site, pretty much ripped straight from their build-a-blog guide. Some of the more interactive components are written in React components and placed on-page with Astro islands. I think it’s got some really cool features, the entire thing is open-source. Favorite features:

Homepage type-greeting (post about how to build your own coming soon!)
Custom Identicons (Evicons)
Recent activity/commits feed
- This has caused me to be slightly more conscious of my commit messages, and how much profanity I use in them (since I know they’ll be more easily viewable on a website that has my name plastered all over it)
Project page, with statuses and links
Fully functioning RSS feed
Comments for posts with utteranc.es

Forehead

This is essentially a game-management and tracking platform for my Golf game. It’s a Svelte/Django application optimized for mobile browser use. I plan on doing a full write-up for Forehead here (2024 State of the Game coming very soon). Favorite features:

Round tracking mode with GPS
Practice modes: differential mode, swing calibration. Much of these practice modes are modeled after techniques developed by Adam Young. I haven’t fully decided if his methods are totally right - but I have been able to measure some success. I may develop new practice modes that model other people’s methods at some point in the future.
Data pipelines and analytics: The start of real analysis and exploration of this data in a much, much larger and meaningful way. Using Pandas + Numpy. More later.

Advent of Code 2024

I really enjoyed AoC this year, though I admittedly had to tap out around Day 14, due to the problems getting too hard and my finals coming due. But overall AoC is almost nothing but upside - I got to spend some time learning idiomatic Python skills and tinker around with my problem solving. My favorite problem/solution combo for the year was probably Day 7, where I used binary and ternary math to come up with individual combinations when computing operator combinations. Not the fastest possible solution, nor the most elegant, but still a creative idea.

No Scroll

This was the big win for the year, big enough that I actually wanted to write something about it here; I really began to reject the grind, the ingrained habit, of scrolling. After a few years of being an on-again-off-again TikTok user, I finally decided that I can’t keep doing this to myself. If there’s one good thing TikTok did for me, it’s wake me up to the cold hard number that is my daily screen time.

What’s interesting about scroll-apps is how the time-usage tend to cascade as they begin to be eradicated. Delete TikTok, and does screen time suddenly drop? Nope - instead the habit simply cascades to the next best app. So then YouTube goes in the bin, what happens next? Instagram usage spikes. Eventually I had to annihilate every scrolling-app from my phone.

And then something really strange happened: unable to cope without the now hard-wired need for scrolling, I began to scroll on other, non-scrolling type apps. I started checking my credit card statements 50 times a day. An uncontrollable compulsion to see what new homes and condominiums had popped up on Zillow in the last ten minutes formed, despite the fact that I’m not currently in the process of moving. The urge to find the satisfaction I once got scrolling Reddit metamorphized into whatever it could most easily latch onto.

This did pass, once my brain realized there was no dopamine to be found even within the depths of my spam folder. Then a new fight began: dealing with all the built up anxiety that I never dealt with because I was using my phone as a numbing agent. I’ll probably write a full post about this at some point in the future because it is genuinely interesting (I also am not sure I completely understand the effects it’s had yet), but for months I thought I was literally dying. My subconscious took the phone-shaped hole in my head and filled it with hypochondriasis. Physically, I’m fine, but it took some time to convince myself of that. I have a whole… philosophy on this stuff now. And I’m not trying to be “Anxious Generation”-pilled, or anything like that, cause I think there’s nuance. Maybe in another post, after this thought crystallizes enough.

Media

I broke my foot in April, which was the worst. Unable to do anything I wanted for a grueling six weeks, I settled for Helldivers 2. I don’t play a lot of games, so take this with a healthy grain of salt, but it’s really something special. That month and a half was a lot more tolerable than it should have been.

In terms of reading - I read some really good stuff this year. Favorites:

Breakfast of Champions by Kurt Vonnegut - funny and introspective. My favorite novel I read this year.
The Verge by Patrick Wyman - An anthological explanation of the emerging world told through famous and not-so-famous characters who lived through the time period of 1480 - 1520. It’s a dense, fairly dry read, but I think Wyman gives it enough color and clarity that you never really lose track of what’s going on or how we’re affected today by the events and ideas he presents.
The Undoing Project by Michael Lewis - Half biography, half psychological and behavioral economics primer, focused on the lives and friendship of Amos Tversky and Danny Khaneman. This is one of those books that might be, admittedly, a little too smart for me, but I can’t help but love the story and the ideas discussed throughout.

Conclusion

2024 was busy. When I was trying to justify going back to school, I kept telling people in my life (and myself) that the reason I was doing it now was because I was never going to be less busy than I am now. That has turned out to be quite the self-fulfilling prophecy. My days feel shorter than ever, but they also feel more rewarding. I think I probably sleep a bit worse than I used to, but that’s probably just something I should get used to. I worry a lot, about my life, about the future and the uncertainty it holds. I also know that I’m doing everything I can, so worrying is futile and unhealthy, but I continue to do it anyways. That’s just who I am. That’s always who I’m going to be. The new year, I hope, is a time to continue growing and building skills, avoiding distractions, continuing my education, and advancing my career. Not much more I can do, but strive for that.

Agile Farm

Thu, 28 Nov 2024 00:00:00 GMT

Agile Farm

ALL STORY POINTS ARE EQUAL

BUT SOME STORY POINTS ARE MORE EQUAL THAN OTHERS

A Little Blazor Shillery

Wed, 20 Nov 2024 00:00:00 GMT

A Little Blazor Shillery

Why do we keep doing this to ourselves?

I think we can all agree that Typescript is a good thing. It offers simple improvements to an already useful language, appending features that most devs agree should be standardized, but for reasons of bureacracy and backwards compatibility won’t be (or at least won’t be for a long while). The additive type support is without a doubt a win; it offers a needed restriction to a language that badly suffers from giving its users too much leniency.

There’s a larger point to be made here: developers consider more restrictions on a language to be a good thing. Without a belt, your pants would sag. They’d still perform their necessary function, but they have a limitation. There’s a risk that they may fall down, embarassing yourself at the office Christmas party again. Typescript is a belt. But JavaScript is not a pair of jeans, it’s more like a mesh article. JavaScript is a language which offers a ton of flexibility and freedom to developers, maybe even a little too much. Thus we end up with piles of spaghetti that technically work but are difficult to maintain and tedious to augment. We have built empires on top this language that was written in less than a month. It’s a triumph of engineering, but not one I really think our profession should continue to cling to as the web standard.

This is just one example of a bandaid, in an industry famous for its band-aids. I would argue that any Node framework is a good example of a band-aid, actually a pretty good one! Components? Yes please. Services and dependency injection? Hello! A rich package ecosystem? Don’t mind if I do, just make sure it’s set up to avoid another incident like leftpad. But there’s something all of these front-end frameworks do wrong. They all use JavaScript! And I don’t understand why it has to be the way. Because that’s the way it’s always been done? Because you were just following orders? Nonsense! Node had the right idea when they proposed that an entire application stack be monolingual. But they got it backwards.

It Doesn’t Have To Be Like This

In comes Blazor. It’s a full-stack web development framework written for .NET. It’s pretty good. You write your HTML the same way you’d write a Razor page, but with better template interpolation. You write your control logic in the block below all that. You can componetize code, or make the file a page by adding a @page directive at the top. There’s JavaScript if you want it (when it’s needed, which isn’t often nor is it never) via an interoperability API that is both convenient and simple.

Earlier this year I built Forehead, a golf application, and I chose to do it in Svelte. I’d seen Svelte floating around for a while. I had heard that it borrowed the right things from React while discarding the bad, and I wanted to give myself a new skill in the front end world. It was a mistake! But here’s the really twisted part: I actually really like Svelte. It’s head-and-shoulders above everything else I’ve tried in the Node ecosystem. But it’s not Blazor. It’s not even close.

Building things in Blazor is just easy. Earlier this year, while working on a new project, my counterpart leaned over to me and asked “Why is this so easy?” He had built a modal that executed some critical program logic in just one hour. He strung together a few components from an existing library (I have high praise for Radzen), spent a few minutes using Visual Studio’s debugging tool to work out the kinks, and he was done. It was nothing fancy, but neither was the code. And that’s a good thing. Blazor gave him the tooling he needed to craft something that is not difficult to maintain or tedious to augment - Blazor gave him the ability to create something easily that doesn’t suck.

Advocacy in Action

Like anything else, getting people who have never heard of a new technology to adopt that technology is, in practice, really difficult. I’ve given lots of presentations about Blazor to large groups, demonstrating success my teams have had with it and how nice the development experience can be. The amount of enthusiasm I get from people who are unfamiliar with it is almost exactly 0. No one gives a shit. Frankly, I get it. If I had to listen to some nerd who doesn’t even work for Microsoft shill for a Microsoft offering this hard, I’d probably tune out in the first five minutes.

On the other hand, the response I get from people who do know a little bit about Blazor is genuinely fantastic. Oftentimes they’re disillusioned front-enders, or they’re .NET fanatics who see the potential change as a liberation. There are lots of people who are hungry for a new web technology, if you know where to look for them.

Inertia takes time to build. First you explain something to someone, and they don’t really care, but that knowledge sticks to them slightly. Then maybe they hear about it again later, and their ears perk up, because they understand the reference. Then maybe at some point they go to a conference and someone is there to talk about the subject, or they undertake a quick personal project on their off hours to learn the skill. And then they either like it, or they don’t. My experience with Blazor is that people tend to like it very much, if they ever get to this point. This might be a fine way to grow awareness, but it’s sluggish and ineffective at growing actual honest on-the-ground use.

The only real advocacy I’ve had any success with has been personal action and exerting pressure. Talking people into using it, and then helping them succeed is the only way to generate any real interest in this technology. I’ve had more success this way than any other - people need to see it work, they need to feel that they “get it”. So for every new project, I try and find an angle and pitch. I would suggest other Blazor zealots take this approach as well.

The End of History

It’s easy, in our still young industry, to imagine that we live at the end of history - that the current way of doing things is the best because it’s what we have now. But life is not a meritocracy - sometimes ideas win just because. I’m glad to see that there are other frontiers forming for web development. Every year there are new technologies that make our current JS-rich ecosystem look a little more backwards. Blazor is one example of this, and it’s one that I wish more people would take note of. I’ll keep fighting my little fight for it until it either sucks or gets overtaken by something even better.

On Relearning

Tue, 11 Jun 2024 00:00:00 GMT

On Relearning

As part of my summer prep for grad school, I took it upon myself (at the recommendation of my program) to do some catch-up on my college math. I struggled immensely with math in undergrad. I retook Calc III after getting a D, and just barely squeaked by in linear algebra. I love math, but at the breakneck pace it was presented to me in school I feel like I barely managed to soak any of it in. After graduating, the skillsets I felt like I had developed seemingly evaporated overnight. Concepts like an integral or a linear transformation seemed abstract and strange to me the first time I was taught them, and over the course of my semesters they solidified into ideas I felt like I could understand in a tangible way. Then they fell away, to the backrooms of my memory where I figured the clerical workers who ran my mind would eventually chuck them away for good after a mandated waiting period.

But learning feels different the second time you do it. No matter how long something goes by, no matter how overgrown with brain-matter-ivory a concept may seem, the synapses might still be there. Perhaps tangled up, or covered in debris, but not ephemeral. Relearning an idea feels like realizing something obvious, or even mundane.

This can easily become a source of frustration, especially when the realization seems so obvious. Why does taking a month off from a language seemingly lose me my embedded syntactical skills, and why am I forced to feel stupid when I inevitably give up the game to go find the answer in someone else’s documentation? Why can I never remember the makeup of an acronym which I’ve referenced in emails a half a dozen times this year? Why do I still feel like I can’t write Python, even though it was my first programming language and I started writing code more than 12 years ago?

It’s easy to get caught up in the feedback loop, and shame craves emittance from an internal frustration. But there’s reasons why all of the above are really the case. I don’t remember that acronym because I’ve internalized the meaning into something else that makes sense to me, leaving the original definition to fade into oblivion. I feel like I still don’t know Python because I’ve never actually written Python code in production as part of my career, so my only exposure has been brief personal projects or small scripts (that I struggle to write every time). The sad, makes-you-think-you’re-dumb truth to it is is that anything you’re not constantly hammering into your head has no real reason to stick around.

I try and find comfort in knowing that there’s probably very little I truly know anything at all about in a given moment. The fun part comes from rediscovering the things I’ve already learned, whenever I want, with an ease that never came the first time. It can be meaningful to turn a “Huh?” into a “Duh.”

Introducing Evicons

Sat, 25 May 2024 00:00:00 GMT

Introducing Evicons

How it started

I like GitHub Identicons. They’re kind of charming, you know? Something simple to give your users to distinguish themselves. I kept mine as my main profile picture for a long time.

I also like the idea of building my own Identicons, so I decided to build something for this blog, to distinguish between pages, posts, or projects. You’ve probably seen them floating around on this site (there’s one at the top of this post). I’m pretty proud of how they turned out and thought I’d document the process here.

To each their own

GitHub Identicons are unique to their user, and are calculated via a cryptographic hash of the user’s username (according to this post). I like the cryptography aspect of this, but I think we can use the seed in a more interesting way, and I’ll get to that. The point is, our Identicons should be cryptographically distinct, deterministic based on their input, which we will call the seed. An Evicon should never be shared by more than one seed. To accomplish this, we can do the same thing that GitHub does and pass the seed through a cryptographic hash function. We’ll work with MD5 for this implementation, same as GitHub. For our seed, we’ll use whatever identifying piece of information we have available for the item we want an Evicon associated with. For example, for blog posts like this one, the seed is simply the title of the post.

Fun fact: I learned that GitHub Identicons only use the first fifteen characters of their hash for their image generation. So while it’s practically impossible, one could theoretically find two usernames that share an identicon. We’ll account for this in our own implementation to ensure that each seed actually has no shared Evicons.

A whole lotta noise

I think an animation would be nice, and with our MD5 hash we should have plenty of information we can use to ‘seed’ the animation, at least for some initial state. I’ve opted to use Perlin noise to generate the animation. Perlin noise is a pretty popular algorithm, especially in the game development space, and is most often used to generate noise maps for terrain generation. It has a nice effect to it, so I built my own implementation that can accept an input seed and generate a completely unique animation of that seed. I’m not going to walk through all the code for this (full source is available here), but I will document the important parts as well as some struggles I ran into along the way.

The algorithm goes, roughly, like this:

Define a grid. Place vectors at every intersection of the grid lines in random directions. These are our Gradient Vectors. Normalize them.

For each pixel in our image, find the four grid corners that surround it, and draw four more vectors for this pixel, with the origin sitting at its respective corner and its end coordinates laying at the pixel’s position. These are our Offset Vectors.


    function drawPerlinAtPosition(x, y, imageData,  gradientVectors, colors,
                                    gridSegmentPxWidth, gridSegmentPxHeight) {
        // Find grid segment
        var gridSegmentX = Math.floor(x / gridSegmentPxWidth);
        var gridSegmentY = Math.floor(y / gridSegmentPxHeight);

        ...

        var offsetVectors = [
            { i: (x - topLeftPosX) / gridSegmentPxWidth, j: (y - topLeftPosY) / gridSegmentPxHeight },
            { i: (x - topRightPosX) / gridSegmentPxWidth, j: (y - topRightPosY) / gridSegmentPxHeight },
            { i: (x - bottomLeftPosX) / gridSegmentPxWidth, j: (y - bottomLeftPosY) / gridSegmentPxHeight },
            { i: (x - bottomRightPosX) / gridSegmentPxWidth, j: (y - bottomRightPosY) / gridSegmentPxHeight },
        ];

        ...

For each of our offset vectors, find the respective Gradient Vectors and calculate a dot product between the vectors. Store these four dot products in a list.


    let dotProducts = [];
    for (var i = 0; i < offsetVectors.length; i++) {
        let gradient = gradientVectorsForPosition[i];

        let dotProduct = offsetVectors[i].i * gradient.i + offsetVectors[i].j * gradient.j;
        dotProducts.push(dotProduct);
    }

Interpolate the dot products, first along the x-axis, and then once more along the y-axis. This final interpolation is our Perlin noise value at that point.


    // Interpolate x
    var x1 = interpolateDotProducts(dotProducts[0], dotProducts[1], relativeX);
    var x2 = interpolateDotProducts(dotProducts[2], dotProducts[3], relativeX);

    // Interpolate y using the results of x interpolation
    var factor = interpolateDotProducts(x1, x2, relativeY);

    ...

    function interpolateDotProducts(a, b, t) {
        return a + smoothstep(t) * (b - a);
    }

    function smoothstep(t) {
        return t * t * t * (t * (t * 6 - 15) + 10);
    }

Use our perlin noise value and interpolate it between our primary and secondary color to get the final pixel hue. How these colors are chosen will be discussed more later, but just know that I wanted the output to be more than a grayscale map.


    function interpolateColor(color1, color2, factor) {
        const result = color1.slice();
        for (let i = 0; i < 3; i++) {
            result[i] = Math.round(result[i] + perlinFactor * (color2[i] - color1[i]));
        }
        return result; // <-- Final hue for pixel
    }

Repeat for all pixels in the image.

Done! Not without some headaches, but I did eventually get an implementation of this working in Vanilla JS, and that’s ultimately what you can see on this site.

Animating

The above algorithm only generates a single perlin noise image, but we want an animation of that noise. We can loop two additional steps to this algorithm to achieve the animation:

Rotate each Gradient Vector slightly.


    function rotateGradientVectors(gradientVectors) {
        for (let y = 0; y <= NUMBER_OF_SEGMENTS; y++) {
            for (let x = 0; x <= NUMBER_OF_SEGMENTS; x++) {
                let gradient = gradientVectors[`${x}:${y}`];
                let angle = Math.atan2(gradient.j, gradient.i); // Get the current angle of the gradient
                let newAngle = angle + Math.sin(TIME_INCREMENT); // Increment the angle by t for rotation

                gradientVectors[`${x}:${y}`] = {
                    i: Math.cos(newAngle),
                    j: Math.sin(newAngle),
                };
            }
        }
    }

Re-render all pixels.

The re-calculation of the gradient vectors will cause the pixel hue to be recalculated just slightly. As the vectors continue to rotate, they’ll approach their initially set positions, and then the animation begins from the beginning.

What about the seed value?

There’s two ways that the inputted seed value affect the animation: defining the colors and defining the initial gradient vectors.

In step #1 I mentioned that the Gradient Vectors should be assigned to a “random” direction, but we already have a random value! We can simply use the MD5 hash of our seed to get the i and j values for each vector, and continue looping through the characters of the hash 2 at a time to build a list of vectors.

Two characters of our hash string will yield two digits of a base-16 number, which translates to a number in decimal ranging between 0 and 256. That gives us plenty of range of direction, but what if we want some of our vectors to be pointing in negative directions? To do this we can establish certain chosen indices using a static list of numbers that are the same every time. Then, to determine which vectors values should be negative, we can find the digits at those indices and mark the value as negative if it is <= 0x08, or positive if it’s above 0x08. I chose the Fibonacci sequence for this list of indices, for no other reason than it’s more interesting than me typing out a random array like [1, 7, 9, 15, …].


    function getFibonacciSequence() {
        let fibonacciSequence = [];
        let n1 = 1, n2 = 2; // Offset Fibonacci so we always get different numbers
        let numberOfSequenceItems = (NUMBER_OF_SEGMENTS + 1) * (NUMBER_OF_SEGMENTS + 1) * 2;
        for (let i = 0; i < numberOfSequenceItems; i++) {
            fibonacciSequence.push(n1);
            let nextTerm = n1 + n2;
            n1 = n2;
            n2 = nextTerm;
        }
        return fibonacciSequence;
    }

    function getRandomDirectionByHash(hash, index) {
        while (index > hash.length) index -= hash.length;
        return parseInt(hash[index], 16) < 8 ? -1 : 1;
    }

And then below, when we build our Gradient Vectors


    for (let y = 0; y <= NUMBER_OF_SEGMENTS; y++) {
        for (let x = 0; x <= NUMBER_OF_SEGMENTS; x++) {
            let originX = x * gridSegmentPxWidth;
            let originY = y * gridSegmentPxHeight;
            let nextSeedX = parseInt(hash.substring(seedIncrement, seedIncrement + 2), 16) *
                getRandomDirectionByHash(hash, fibonacciSequence[directionIncrement++]);
            let nextSeedY = parseInt(hash.substring(seedIncrement + 2, seedIncrement + 4), 16) *
                getRandomDirectionByHash(hash, fibonacciSequence[directionIncrement++]);

    ...

A pop of color

There’s 16 bytes in an MD5 hash, which results in a hexadecimal value with thirty-two digits. We can take the first six digits and the second six digits, and we now have two “random” colors that we can use. So the hash ef8254cee2dab002fdcc623de5da9b23 translates to:

#ef8254 #cee2da

Drawbacks

There’s a few drawbacks to our Perlin Noise approach to the Evicon, (I’ve affectionately dubbed the script EviconsV1.js for a reason) worth going over.

They’re needlessly complicated.
It’s fairly easy to generate an Evicon that resembles another one, so they’re not really all that good at identifying anything, even if they offer a decent approximation.
Perlin Noise has a time complexity of O(2ⁿ), so it’s relatively slow. I haven’t seen much dip in browser performance, but it’s something I’ll have to keep on as the site grows. Simplex noise is an alternative (also developed by Ken Perlin) that might be a better candidate for next time.

Fin

This was fun, I love the way they look, and I think they add a nice bit of visual flair to the site that you won’t see elsewhere. You can mess around with creating your own Evicons here (⬅️ credit to @JVanAuken for the idea to build this page).

An annoying quirk of Copilot and how to quickly get around it

Thu, 23 May 2024 00:00:00 GMT

Sometimes when ChatGPT is down I’ll resort to using Copilot for more ‘generic’ requests. Copilot has more guardrails around it, making it impractical for any work that isn’t software development, so sometimes you’ll run into something like this:

This is a silly example, but Copilot has burned me a couple of times on things like this. Thankfully, there’s an easy way to fight back.


    string meaningOfLove =
        """
            The meaning of love varies among different people and cultures. It's a complex and 
            powerful emotion that can be hard to define as it encompasses a range of feelings and attitudes.
        """;

Not a very good answer, but still. Checkmate, Copilot.

Hello, world

Wed, 22 May 2024 00:00:00 GMT

Hello, world!

I’m starting this blog to document some thoughts and projects on engineering, data science, life, etc. It’s a fun thing I can tinker with on the side as well.

I don’t expect anyone to read any of this stuff, but if you do read something and you find it interesting/useful/funny/expletive please reach out to let me know!

Site built with Astro.