Arpit Mathur is a Principal Engineer at Comcast Labs where he is currently working on a variety of topics including Machine Learning, Affective Computing, and Blockchain applications. Arpit has also worked extensively on Android and iOS applications, Virtual Reality apps as well as with web technologies like JavaScript, HTML and Ruby on Rails. He also spent a couple of years in the User Experience team as a Creative Technologist.
Understanding the difference between find_or_create_by and create_or_find_by: For the project at work right now we were debating the best way to approach the “find user unless they already exist” implementation (we are using a Node based framework which we have to customize since our service is checking other services to validate the existence of a user). When in doubt, I try to see how Rails does it and I actually did not know it had two different implementations of the requirement based on whether the accidental creation of duplicates via a race condition was acceptable or not. Interesting read.
The Effective Engineer (YouTube): Originally published 8 years ago, still a great watch. Edmond Lau talks about 5 approaches to work that would make you a more effective engineer: 1. Optimize for learning, 2. Invest in iteration speed, 3. Validate your ideas aggressively, 4. Minimize operational burden and 5. Building a great engineering culture.
Design
List of UX Research Templates – 12 years ago when I was working in the UX team, I used to be very involved with the UX Research teams, building the prototypes that would be tested with actual users. That said, I wasn’t that involved with the UX Research process. In my current project, we are doing some ad-hoc UX Research. This list of templates could be very useful.
Misc
Inside The New York Times’ Big Bet on Games (Vanity Fair): I spent a part of last year working on some gaming product concepts and so am very interested in how this space is evolving. This piece was an interesting insight into how the games products (Crossword, Spelling Bee, Wordle, Digit, Connection, etc) are run inside the Times. The Times may not be the first company to come to mind when thinking of gaming but interesting to see how games is becoming a meaningful part of the Times’ offerings
Figma’s database migration from one giant DB to multiple vertically partitioned databases – My knowledge of database-level aspects of applications is just about functional, relying more on managed services or backend teams to provide my application level code a robust api to work with. However, with my latest project at work, I am starting to look deeper into database elements just to be aware of the tradeoffs we might need to make in the future. This was a great read on how the Figma team transitioned their giant db to a number of vertically partitioned smaller ones and instead of splitting each table across many databases, moved groups of tables onto their own databases.
Database Fundamentals – A great “from the ground up” read on database technology and concepts. I specially enjoyed the part on various algorithms the database storage engines use.
Design
The Genius Design of Dutch Money (YouTube) – It’s a bummer that these are no longer in circulation with Netherlands moving to the Euro, but the focus on clarity and fun make these some of the most interestingly designed currency notes
Fun
Children of Time by Adrian Tchaikovsky (Book) – Not surprised that this has won numerous awards and is often talked about as one of the best Science Fiction books of all time. Children of Time explores what happens when a terraforming project goes wrong and a virus designed to speed up simian evolution accidentally ends up evolving spiders instead. I really liked the exploration of civilized spider societies and how their technology would be based on very different primitives than humans’.
In trying to write this post, I was looking back at my retrospectives for the last few years and once again struck by how different every year has been from the previous year. 2023 continued that trend, but had a different note to it: at the end of most previous years I felt I had learned a lot, specially around technology; 2023 was a great year where I was finally able to apply a lot of those learnings into building something impactful.
Selling an idea
In my wrap up to 2022, I had mentioned that I was starting to focus on 2 things: Web3 and building a small team to explore new ideas in that space. The biggest win of 2023 was building momentum around a new idea there.
Presenting the idea at an internal "What's Next" event to 5000 folks was quite an experience
Getting buy-in in a large organization/corporation is hard and it was a really educational experience getting various leaders excited about a very different product opportunity and commit financially to it. I have been thinking about why this project was better received than other pitches earlier and some thoughts come to mind:
Quick prototype: We showed the core idea in a prototype that was assembled in a week. The short time helped us to focus only on the core concept and not build other elements that were nice to have but only muddied up the narrative.
Cross-organization team: We build the concept in partnership with the teams from Universal Pictures and the cross team collaboration helped with the credibility of the idea
Adaptibility: The idea we talked about could be adapted to various goals from commerce to engagement. The really small prototype allowed folks to fill in the blanks of how this could work in their own minds.
Timing: Not so much a credit to anything we did, but in general there was a bigger appetite for new ideas across the organization.
Probably the last one I’d add was my own personal belief in the idea and not purely from a practical sense. One of my favorite reads of all time was the book Creativity Inc which talked about how Pixar grew to the behemoth it is today. In that, Ed Catmull (former CEO) writes about how new ideas are often “ugly babies” – they have a germ of something that could be huge but aren’t quite there yet – and have to be protected against the “hungry monster” that is the rest of the org that is generating all the revenue but is always hungry for more resources (financial and attention). Ugly babies need believers and I definitely am for this idea (which meant I did create more presentations this year than ever before 🤣)
From concept to release
Taking an idea to launch was an experience I hadn’t had in a long time. I am lucky in landing a team that punches way above their weight and can balance the exploratory nature of our app with the needs of a production app (scale, security, privacy, etc). Given we were still fairly small team compared to the ambitions of this project here is how we approached the project:
Agile-ish: We are basically using Kanban process for all engineering (most of the design came from an agency that worked out of band). We don’t have sprints yet since the exact features for the app aren’t set yet. There was a lot of creative chaos in 2023 but I am hoping that it settles to something more traditional in 2024.
Build less: We lean towards using managed services and open-source products when possible. Integrating those services is a enough work to keep us busy anyway
Monolith over microservices: One of the thoughts I remember reading somewhere was that microservices are meant to solve organizational problems, not technical ones. Between that, and DHH’s article that came out around the time we were planning the architecture has meant that we are on team-monolith till we grow.
I’ll share more about other technical aspects in a future post.
And while the app is *in production* right now, its still a beta so I’ll share the link once we open it up a little more.
Misc
This post has already gotten pretty long, so some quick updates below:
Went to India for the first time in 5 years, first time with Dana and that was a lot of fun. I hadn’t seen Diwali in India since I moved to the US 20 years ago so that was amazing
Read a few books but dropped out of a lot before finishing them which is unusual for me. How not to be wrong was probably my most interesting read. I have been reading a lot of graphic novels which have been fun. Strongly recommend “Its lonely at the center of the Earth“.
Finally, on a sad note, we lost 3 of my feline buddies this year. Grayling, Magical and Fuzzy were the friendliest and most loving of my cats and losing them in the first half of the year one after another was devastating. I miss them a lot but I like to think they are around me even now. They were such good boys.
Hug your pets if you have them and support local rescues and animal/nature/wildlife orgs if you can
The latest project I am working on requires us to set up a number of cloud services on AWS. While I have some experience doing that on Google Cloud Platform, the AWS system and tooling are a learning experience.
This week we are setting up a number of AWS Lambdas. We are trying to do this right from day one by enabling unit/integration tests and automated build systems from the start. We also made the choice to start immediately with TypeScript rather than migrate to it over time as we have done in other projects.
I tried using the AWS SAM CLI for creating the Lambdas today. There were a couple of hiccups that I wanted to share here.
1. Typescript templates are only available in ZIP-based deployments
I was glad to see that the SAM CLI allowed for the creation of TypeScript based projects out of the box. What took me a long time to realize was that the TypeScript templates only appear as options if you use the ZIP-based package type. I tried various combinations of node versions, etc before trying the zip based option which finally worked. There are also way too many tutorials on the web that start you out with a JS based project and then you add a number of config changes to migrate to TS which is a pain.
2. Colima-based hiccups
Most of the tutorials I found using SAM required you to use Docker Desktop, which unfortunately is no longer an option at work given their new licensing terms.
Thankfully this gist helped set up a local environment using Colima that seemed to work at first.
The only problem was that trying to test the code locally did not work as the local connections to the container would timeout without being able to connect. This turned out to be caused by Colima not enabling reachable IP addresses by default for performance reasons. Restarting Colima with colima start --network-address fixed this but I only discovered this after a couple of hours of banging my head against the computer.
Ah well, things are working right now and I need to get home 👋
This week the GDG-Philadelphia group that I manage collaborated with the Philly Cocoa group to run our first in-person event since Covid. The number of folks who showed up was much higher than I thought they would, I think everyone just really wanted to meet each other IRL again.
The developer meetups in the city have unfortunately not re-emerged from Covid as much and I am hoping they do. I am hopeful that more such events start happening again.
Thanks to Kotaro for leading so much of the organization effort for this event, and to Comcast Labs and the Lift Labs group for sponsoring the space and food.
I spent most of last week in Denver attending EthDenver 2023, one of the biggest Ethereum-centric conferences in North America. This was my first EthDenver event (first of many I hope) and I wasn’t sure what to expect given the roller coaster that almost seems a constant in this domain. That said, the five days I spent there were a great way to get a glimpse into various projects and themes in play in that ecosystem. This post covers some of the projects and ideas I saw there.
A multi-chain future
I attended Axelar’s mini-summit on interoperability which was very interesting. The tl;dr of various panel discussions was that while we might have our L1 or L2 chains of choice, we are living in a multi-chain world, and interesting applications will appear in a lot of different chains. That said, at least so far, Ethereum remains the main chain of choice acting as a hub with spokes out to various other chains. There were also some interesting discussions on the safety of bridges that connect various chains and which have been in the news a number of times for getting hacked. I am not sure knowledgable about a lot of the chains that were mentioned but found the discussions fascinating and would like some time to explore Axelar, Polkadot and Cosmos.
The Social Web3
I ended up attending various sessions on Web3 based social platforms, and there were a lot of those around. The ones I did find most interesting included Huddle01 (Web3 based Zoom), Sending.me (Web3 based Discord), and Dispatch, a company building NFT based shopping tools as well as “engagement NFTs” that encode interactive experiences like polls etc. More interesting though was learning about a number of emerging protocols for Web3 based communication like DMTP, XMTP and Lens. I am really curious about Lens as it comes from the company behind the successful Aave protocol and has some very interesting primitives, like using NFTs to define user connections which I do think is kinda interesting.
Incentive Design and Economics
Thanks to my friend Mike Mccoy at the CryptoEconLab, I was able to attend their day long mini-summit on cryptoeconomics. I’d say I understood maybe 40% of the ideas shared here, but the talks were very good. I specially enjoyed the session on Protocol Gas Dynamics and MEV, Quadratic Funding by GitCoin and on NFT Auction Design by Tarun Chitra. You can find all the talks at the event on their YouTube channel
Miscellaneous Thoughts
There were a lot of sessions on privacy and zero-knowledge proofs. Not surprising since privacy is a big deal these days and ZK technology seems a promising space.
There were more new wallet startups I saw at the event than I was expecting. MetaMask remains the big name there and their announcement about Snaps, their extension framework, was awesome, but it was interesting to see a number of wallet providers focusing on user experience.
Unity’s announcement about their partnerships with a number of major Web3 platforms happened around EthDenver. Hopefully this results in some compelling Web3 games and not just standard games with Web3 bolted on
Although announced last year, I only heard of Instagram’s new age verification tests this morning. In order to verify that you are above 18, Instagram will require users to choose one of the three methods of verification: a government-issued ID, a video selfie that an AI tool can then review, or a social vouching system where three of your followers can confirm you are over 18.
Social Vouching: This option allows you to ask mutual followers to confirm how old you are. The person vouching must be at least 18 years old, must not be vouching for anyone else at that time and will need to meet other safeguards we have in place. The three people you select to vouch for you will receive a request to confirm your age and will need to respond within three days.
The social vouching system to me is the most interesting, especially since I have been noodling over the idea of decentralized identity and verifications for a while. The system closely resembles the idea of Solidarity Lending which I first heard about when I read about Mohammad Yunus‘s Nobel Prize-winning work with the Grameen Bank.
Web3 applications take this a step further with “M-of-N” (multisig) transactions where the “N” is the pre-defined community of individuals that collectively decide on how transactions should be carried out.
Multisig transactions are also referred to as M-of-N transactions, with M being the required number of signatures or keys and N being the total number of signatures or keys involved in the transaction.
The idea of a collective making confirming a claim is a core principle of decentralized systems and it is encouraging to see it starting to emerge in mainstream applications. Could we ever make a sign-up/sign-in system based on collective claims as mainstream and the cryptic oAuth flows we have today with singular entities verifying claims? Here’s hoping 🤞 !
One of my most memorable/educational meetups was one by Philly.rb on audit logging. Audit logs record the occurrence of an event, the time at which it occurred, and any other useful data. The insight I remember from that talk was that audit logs are generated by creating a record for every change event in a new table. For example, your project may have a model User and a model ChangeOperation (usually generated by some library) and every time any data on the user model is changed, a new ChangeOperation instance is created and persisted in the database. You can later view all the ChangeOperation s the User model has gone through.
I am at EthDenver this week and it’s been great. There are some really interesting new projects using NFTs in a number of new ways. A typical NFT is such a simple idea that it is very amenable to representing different properties of a product. The Lens Protocol is one such interesting project that wants to represent social graphs on the blockchain. Like a few other social-network-on-the-chain projects, Lens uses NFTs to represent user identities.
Where Lens does something interesting though is representing verbs like “following” as an NFT. From their documentation:
Profile NFTs contain a FollowModule. This module contains the logic that allows different accounts to be issued Follow NFTs to record their relationship to the main profile on-chain.
Extracting the follow action to its own smart contract is an interesting move and allows you to create all sorts of various kinds of follows like “create a follow NFT only if the user has this other NFT” or “create a follow NFT if they pay a certain amount” (basically premium follows).
Definitely an interesting idea and something I need to dive into a bit more.
It’s almost mid-February, so I am pretty late with this, but better late than never I guess. I do enjoy writing my yearly retrospectives (you can see my previous ones here) since they give me an opportunity to pause for a second and reflect on the work done, something I usually don’t do as I am always running to the next thing I want to work on.
Web3 and NFTs
My retrospectives for the last few years have begun with the myriad of technologies I have learned that year, from VR/AR, to chatbots and Machine Learning. This year was a bit different though. I finally got to pursue a technology that I have been passionate about for a long time (and have written about a number of times previously) but have had a hard time finding a project relevant to my company: Blockchain and Web3.
In 2022 I spent most of my time developing and pitching a Web3 idea that resonated well internally. I am really excited that the effort is ramping up in 2023. The Web3 space is a curious one and is often a target for ridicule but the potential for creating experiences where end users are not just passive consumers but active participants in the evolution of an experience is pretty exciting. I wrote some of my thoughts on why NFTs are an exciting space and worthy of a second look, but even if it doesn’t immediately convert you into a believer, at least be aware that a lot of very smart people are working with more noble goals than get-rich-quick schemes.
Building Teams
My role has often been at the early stage of products that then ramps up to a full production team. This year I did quite a bit of that as well, ramping up the UI team for a soon-to-be-announced product. The role involved helping migrate an internal prototype to production quality code as well as hiring a team that would take it forward.
While I have interviewed candidates for small teams before, this was for a larger team with folks being hired for roles at various levels. I am pretty proud of the team we finally built but I do want to grow my own internal framework for assessing engineers and managers.
I have seen 2 approaches to hiring:
Have a loose idea of the skills you are hiring for and then find the candidates that are close to those. If the matches aren’t close, still hire for the skills you find.
Have a very exact idea of what you are looking for and seek out those who match that very strictly
Most approaches fall somewhere in the middle of those 2 extremes but sometimes we lean too heavily on #1, especially when we are in a rush to hire candidates. One approach I found useful was to explicitly list the technical and cultural skills I was looking for in an Airtable table, develop a set of questions that would reveal those ahead of time, and use the table to explicitly score the candidates. It worked well for the most part but I have a lot more to learn myself in that role.
Lightning JS
For both the unannounced product and our explorations of Web3, I spent a lot of time working on TV UIs using Comcast’s open-source Lightning JS UI framework. LUI is a JavaScript-to-Canvas/WebGL framework that skips the HTML DOM completely. The approach makes it a lot more performant on low-memory devices like TVs and Set-top boxes. While the approach is great, some of the patterns in LUI are very different compared to other UI frameworks I have worked with. I am pretty competent with LUI now but it was an adventure to get there.
An interesting approach at work is that higher-order UI components are now being built by a team reporting to the design/UX leadership. This has made the components very faithful to the visual design patterns that the UX team imagines, though I do think there are optimizations at the performance level that need to happen since we do hit weird jank issues on some of the devices we work with.
LUI is focused on TV-based experiences but might be fun to try some mobile/desktop-optimized apps/games that don’t need the native UI frameworks.
Flutter
In my off-time, I still enjoy working in Flutter. This year’s fun side project was a grocery-tracking app I named Grocery Timer. GT has not revolutionized the world of grocery management but has still been a fun playground to try different UIs for a surprisingly complicated problem space. Keeping things simple so that data entry isn’t a chore while still providing value and the right reminders has been a challenge. That said, I personally find the app very useful and so its relative obscurity on the app stores doesn’t hurt too much 🙂
Books
Didn’t do as much reading as usual this year but did find some very good reads, specially Loonshots and The Courage to be Disliked. Highly recommended. I am also rediscovering the joy of graphic novels and YouTube channels, especially For the love of Comics, has been a great source of great recommndations.
I didn’t write as much in 2022, at least not publically, but that is one of my resolutions for 2023. Stay tuned 📻
I finally watched this YouTube talk with the writers of the Expanse series, one of my favorite Sci-fi series of all time (mostly through the TV show but I did end up reading 2 of the 3 books beyond the TV series). The video is from 8 years ago but the ideas shared were pretty timeless. My best friend and I have talked for years about writing a sci-fi book together but have never gotten past a few ideation sessions (too many personal projects 🙂 )
Some interesting notes from the talk:
Birth of the project: The work started as an MMORPG game that didn’t happen, which then resulted in a tabletop game and then finally the book. The book was more of a “we have all this world already created” moment than a fresh attempt at a book.
Collaboration The two writers (yeah I thought it was one writer but turns out James Corey was just a pen name) worked together to create an outline of events and then wrote alternate chapters (each focusing on the events of their own character) and then gave it to each other to edit. Creative collaboration is hard and this seemed like a great way to do it.
On creating Sci-fi stories: Unlike other genres, sci-fi doesn’t have a default plot type and you can bring in other genres. So book 1 is “sci-fi noir”, other books are “sci-fi westerns”, “sci-fi-political thrillers” etc
On why Fantasy seems more popular than Science Fiction Interesting perspective on why sci-fi kinda lost to fantasy: Sci-fi became more focused on the science and math aspects of their ideas and telling really depressing stories to a point that the authors were mostly writing to impress each other. Fantasy stories focused more on the characters and plot and were a lot more upbeat.
Why do creative collaborations fail? One of the many reasons collaborations can fail is because the collaborators may have different ideas about the work (the kind of story or message they are sharing). The alignment on the world early on helped steward the effort towards success
On the limits of “Hard Sci-fi” The writers don’t consider Expanse “Hard Sci-fi”, they just got a lot of credibility by keeping the stories anchored in general scientific ideas. That said, they never let the scientific aspects limit their creative work. Also keeping the limitations of science made for interesting plots that couldn’t be just solved with miraculous technology
Code Zen 