Arpit Mathur is a Principal Engineer at Comcast Labs where he is currently working on a variety of topics including Machine Learning, Affective Computing, and Blockchain applications. Arpit has also worked extensively on Android and iOS applications, Virtual Reality apps as well as with web technologies like JavaScript, HTML and Ruby on Rails. He also spent a couple of years in the User Experience team as a Creative Technologist.
Enabling branch deployments through IssueOps with GitHub Actions (Github.com) – This week there was a lot of debate on the right model we need to adopt for shipping software. While the first couple of production deploys have followed the traditional merge “deploy main at end of sprint” model, Github’s “push to prod on merge request” is an interesting approach. I have the draft of a post on my thoughts on it that I’ll share soon.
The Google Wallet Challenge (YouTube.com): A dev challenge to build something with Google Wallet is fine, but more interesting: I didn’t realize you can push arbitrary data, including custom collectibles, into the wallet (Sample code)
The myths we tell ourselves about American farming (vox.com): An insightful, though depressing read, on Agricultural Exceptionalism, or how our romanticized notions of farmers and farming are allowing that industry to operate under a different set of environmental and labor laws that hurt our society.
One of the fairly new APIs added to the web stack is the Navigator.share API. The API allows websites and apps to invoke the native share sheet on OS running the web browser to let the user share text, links and images to the services they actually use.
This week, one of the tasks on our new web app (written in Flutter) was to transition from our custom “share this link” UI that was preconfigured to a select set of services (Twitter, Facebook, Reddit, email) to the native share api allowing users to share the link on their configured services.
Some thoughts on the Share sheet
Before I continue, lets talk about the OS share system in general. If there was ever a “God Object” equivalent of user interfaces, the system share sheet on mobile devices has to be it. The widget has become sort of like an escape hatch for the underlying application with not only share capabilities, but a host of other actions in the same menu (Find on page? Print? Add to home screen?) which keeps getting updated by your installed applications. I don’t think anyone can list the menu options there from memory, and yet if we expect to run any action on any data from the underlying application, we expect that action to be available in the share sheet.
I bring this up because the native share from a webpage is now not as simple as sending a title, text, link and image to the share sheet and assuming it would work. Different behaviors from different apps get enabled based on various combination of the 4 fields. For example, one service we wanted to make sure users could use was Instagram (assuming they were Instagram users). IG behavior via share is something like this:
Input
Instagram behavior
Only title, or title + text
Cannot share to Instagram
link + any combination of title and text
Share with a friend on Instagram as a message
Only image
Share that image in a post, story or message
Image + any of the other fields
Share image as a reel
Instagram behavior matrix
Other image oriented services like Snapchat and Pinterest also seem to be sensitive to the combination of the 4 fields sent. Its kind of a bummer that we have to now spend QA cycles making sure we are sending the right order of fields to get the desired behavior from the receiving app.
Native share on the desktop?
The other challenge to native sharing is that while the browser API may be available on a lot of platforms, native share on the desktop does not make sense since most people don’t install apps for the services they use on their desktop which means that the share menu is pretty barren. Chrome on the Mac desktop, perhaps sensibly, does not support the web share API but Safari does.
Our solution for the app now is to use our custom share UI on desktop browsers but use the native one on mobile. Feels a little hack-ish but it makes the most sense from a user’s point of view.
Sharing images on Flutter web apps with Share Plus
The last aspect of the implementation that tripped us a bit was trying to include images from a web URL along with the title and link to the share API. We are using the Share Plus library which is pretty good and supports the native share API. Unfortunately we didn’t find any links on how to convert an image link to the image attachment (its still early days for Flutter web I suppose). If anyone else needs the snippet, here is how you do it:
final http.Response responseData =
await http.get(Uri.parse(linkToFile));
Uint8List bytes = responseData.bodyBytes;
var file = XFile.fromData( bytes, mimeType: 'image/png', name: "MyPic.png");
await Share.shareXFiles([file], text: 'Hello', subject: 'Check out this image' );
The thing that I wasted 2 hours on today was this snippet not working cause the name I was giving my file did not include the file extension 🤦♂️. So be warned 🙂
Understanding the difference between find_or_create_by and create_or_find_by: For the project at work right now we were debating the best way to approach the “find user unless they already exist” implementation (we are using a Node based framework which we have to customize since our service is checking other services to validate the existence of a user). When in doubt, I try to see how Rails does it and I actually did not know it had two different implementations of the requirement based on whether the accidental creation of duplicates via a race condition was acceptable or not. Interesting read.
The Effective Engineer (YouTube): Originally published 8 years ago, still a great watch. Edmond Lau talks about 5 approaches to work that would make you a more effective engineer: 1. Optimize for learning, 2. Invest in iteration speed, 3. Validate your ideas aggressively, 4. Minimize operational burden and 5. Building a great engineering culture.
Design
List of UX Research Templates – 12 years ago when I was working in the UX team, I used to be very involved with the UX Research teams, building the prototypes that would be tested with actual users. That said, I wasn’t that involved with the UX Research process. In my current project, we are doing some ad-hoc UX Research. This list of templates could be very useful.
Misc
Inside The New York Times’ Big Bet on Games (Vanity Fair): I spent a part of last year working on some gaming product concepts and so am very interested in how this space is evolving. This piece was an interesting insight into how the games products (Crossword, Spelling Bee, Wordle, Digit, Connection, etc) are run inside the Times. The Times may not be the first company to come to mind when thinking of gaming but interesting to see how games is becoming a meaningful part of the Times’ offerings
Figma’s database migration from one giant DB to multiple vertically partitioned databases – My knowledge of database-level aspects of applications is just about functional, relying more on managed services or backend teams to provide my application level code a robust api to work with. However, with my latest project at work, I am starting to look deeper into database elements just to be aware of the tradeoffs we might need to make in the future. This was a great read on how the Figma team transitioned their giant db to a number of vertically partitioned smaller ones and instead of splitting each table across many databases, moved groups of tables onto their own databases.
Database Fundamentals – A great “from the ground up” read on database technology and concepts. I specially enjoyed the part on various algorithms the database storage engines use.
Design
The Genius Design of Dutch Money (YouTube) – It’s a bummer that these are no longer in circulation with Netherlands moving to the Euro, but the focus on clarity and fun make these some of the most interestingly designed currency notes
Fun
Children of Time by Adrian Tchaikovsky (Book) – Not surprised that this has won numerous awards and is often talked about as one of the best Science Fiction books of all time. Children of Time explores what happens when a terraforming project goes wrong and a virus designed to speed up simian evolution accidentally ends up evolving spiders instead. I really liked the exploration of civilized spider societies and how their technology would be based on very different primitives than humans’.
In trying to write this post, I was looking back at my retrospectives for the last few years and once again struck by how different every year has been from the previous year. 2023 continued that trend, but had a different note to it: at the end of most previous years I felt I had learned a lot, specially around technology; 2023 was a great year where I was finally able to apply a lot of those learnings into building something impactful.
Selling an idea
In my wrap up to 2022, I had mentioned that I was starting to focus on 2 things: Web3 and building a small team to explore new ideas in that space. The biggest win of 2023 was building momentum around a new idea there.
Presenting the idea at an internal "What's Next" event to 5000 folks was quite an experience
Getting buy-in in a large organization/corporation is hard and it was a really educational experience getting various leaders excited about a very different product opportunity and commit financially to it. I have been thinking about why this project was better received than other pitches earlier and some thoughts come to mind:
Quick prototype: We showed the core idea in a prototype that was assembled in a week. The short time helped us to focus only on the core concept and not build other elements that were nice to have but only muddied up the narrative.
Cross-organization team: We build the concept in partnership with the teams from Universal Pictures and the cross team collaboration helped with the credibility of the idea
Adaptibility: The idea we talked about could be adapted to various goals from commerce to engagement. The really small prototype allowed folks to fill in the blanks of how this could work in their own minds.
Timing: Not so much a credit to anything we did, but in general there was a bigger appetite for new ideas across the organization.
Probably the last one I’d add was my own personal belief in the idea and not purely from a practical sense. One of my favorite reads of all time was the book Creativity Inc which talked about how Pixar grew to the behemoth it is today. In that, Ed Catmull (former CEO) writes about how new ideas are often “ugly babies” – they have a germ of something that could be huge but aren’t quite there yet – and have to be protected against the “hungry monster” that is the rest of the org that is generating all the revenue but is always hungry for more resources (financial and attention). Ugly babies need believers and I definitely am for this idea (which meant I did create more presentations this year than ever before 🤣)
From concept to release
Taking an idea to launch was an experience I hadn’t had in a long time. I am lucky in landing a team that punches way above their weight and can balance the exploratory nature of our app with the needs of a production app (scale, security, privacy, etc). Given we were still fairly small team compared to the ambitions of this project here is how we approached the project:
Agile-ish: We are basically using Kanban process for all engineering (most of the design came from an agency that worked out of band). We don’t have sprints yet since the exact features for the app aren’t set yet. There was a lot of creative chaos in 2023 but I am hoping that it settles to something more traditional in 2024.
Build less: We lean towards using managed services and open-source products when possible. Integrating those services is a enough work to keep us busy anyway
Monolith over microservices: One of the thoughts I remember reading somewhere was that microservices are meant to solve organizational problems, not technical ones. Between that, and DHH’s article that came out around the time we were planning the architecture has meant that we are on team-monolith till we grow.
I’ll share more about other technical aspects in a future post.
And while the app is *in production* right now, its still a beta so I’ll share the link once we open it up a little more.
Misc
This post has already gotten pretty long, so some quick updates below:
Went to India for the first time in 5 years, first time with Dana and that was a lot of fun. I hadn’t seen Diwali in India since I moved to the US 20 years ago so that was amazing
Read a few books but dropped out of a lot before finishing them which is unusual for me. How not to be wrong was probably my most interesting read. I have been reading a lot of graphic novels which have been fun. Strongly recommend “Its lonely at the center of the Earth“.
Finally, on a sad note, we lost 3 of my feline buddies this year. Grayling, Magical and Fuzzy were the friendliest and most loving of my cats and losing them in the first half of the year one after another was devastating. I miss them a lot but I like to think they are around me even now. They were such good boys.
Hug your pets if you have them and support local rescues and animal/nature/wildlife orgs if you can
The latest project I am working on requires us to set up a number of cloud services on AWS. While I have some experience doing that on Google Cloud Platform, the AWS system and tooling are a learning experience.
This week we are setting up a number of AWS Lambdas. We are trying to do this right from day one by enabling unit/integration tests and automated build systems from the start. We also made the choice to start immediately with TypeScript rather than migrate to it over time as we have done in other projects.
I tried using the AWS SAM CLI for creating the Lambdas today. There were a couple of hiccups that I wanted to share here.
1. Typescript templates are only available in ZIP-based deployments
I was glad to see that the SAM CLI allowed for the creation of TypeScript based projects out of the box. What took me a long time to realize was that the TypeScript templates only appear as options if you use the ZIP-based package type. I tried various combinations of node versions, etc before trying the zip based option which finally worked. There are also way too many tutorials on the web that start you out with a JS based project and then you add a number of config changes to migrate to TS which is a pain.
2. Colima-based hiccups
Most of the tutorials I found using SAM required you to use Docker Desktop, which unfortunately is no longer an option at work given their new licensing terms.
Thankfully this gist helped set up a local environment using Colima that seemed to work at first.
The only problem was that trying to test the code locally did not work as the local connections to the container would timeout without being able to connect. This turned out to be caused by Colima not enabling reachable IP addresses by default for performance reasons. Restarting Colima with colima start --network-address fixed this but I only discovered this after a couple of hours of banging my head against the computer.
Ah well, things are working right now and I need to get home 👋
This week the GDG-Philadelphia group that I manage collaborated with the Philly Cocoa group to run our first in-person event since Covid. The number of folks who showed up was much higher than I thought they would, I think everyone just really wanted to meet each other IRL again.
The developer meetups in the city have unfortunately not re-emerged from Covid as much and I am hoping they do. I am hopeful that more such events start happening again.
Thanks to Kotaro for leading so much of the organization effort for this event, and to Comcast Labs and the Lift Labs group for sponsoring the space and food.
I spent most of last week in Denver attending EthDenver 2023, one of the biggest Ethereum-centric conferences in North America. This was my first EthDenver event (first of many I hope) and I wasn’t sure what to expect given the roller coaster that almost seems a constant in this domain. That said, the five days I spent there were a great way to get a glimpse into various projects and themes in play in that ecosystem. This post covers some of the projects and ideas I saw there.
A multi-chain future
I attended Axelar’s mini-summit on interoperability which was very interesting. The tl;dr of various panel discussions was that while we might have our L1 or L2 chains of choice, we are living in a multi-chain world, and interesting applications will appear in a lot of different chains. That said, at least so far, Ethereum remains the main chain of choice acting as a hub with spokes out to various other chains. There were also some interesting discussions on the safety of bridges that connect various chains and which have been in the news a number of times for getting hacked. I am not sure knowledgable about a lot of the chains that were mentioned but found the discussions fascinating and would like some time to explore Axelar, Polkadot and Cosmos.
The Social Web3
I ended up attending various sessions on Web3 based social platforms, and there were a lot of those around. The ones I did find most interesting included Huddle01 (Web3 based Zoom), Sending.me (Web3 based Discord), and Dispatch, a company building NFT based shopping tools as well as “engagement NFTs” that encode interactive experiences like polls etc. More interesting though was learning about a number of emerging protocols for Web3 based communication like DMTP, XMTP and Lens. I am really curious about Lens as it comes from the company behind the successful Aave protocol and has some very interesting primitives, like using NFTs to define user connections which I do think is kinda interesting.
Incentive Design and Economics
Thanks to my friend Mike Mccoy at the CryptoEconLab, I was able to attend their day long mini-summit on cryptoeconomics. I’d say I understood maybe 40% of the ideas shared here, but the talks were very good. I specially enjoyed the session on Protocol Gas Dynamics and MEV, Quadratic Funding by GitCoin and on NFT Auction Design by Tarun Chitra. You can find all the talks at the event on their YouTube channel
Miscellaneous Thoughts
There were a lot of sessions on privacy and zero-knowledge proofs. Not surprising since privacy is a big deal these days and ZK technology seems a promising space.
There were more new wallet startups I saw at the event than I was expecting. MetaMask remains the big name there and their announcement about Snaps, their extension framework, was awesome, but it was interesting to see a number of wallet providers focusing on user experience.
Unity’s announcement about their partnerships with a number of major Web3 platforms happened around EthDenver. Hopefully this results in some compelling Web3 games and not just standard games with Web3 bolted on
Although announced last year, I only heard of Instagram’s new age verification tests this morning. In order to verify that you are above 18, Instagram will require users to choose one of the three methods of verification: a government-issued ID, a video selfie that an AI tool can then review, or a social vouching system where three of your followers can confirm you are over 18.
Social Vouching: This option allows you to ask mutual followers to confirm how old you are. The person vouching must be at least 18 years old, must not be vouching for anyone else at that time and will need to meet other safeguards we have in place. The three people you select to vouch for you will receive a request to confirm your age and will need to respond within three days.
The social vouching system to me is the most interesting, especially since I have been noodling over the idea of decentralized identity and verifications for a while. The system closely resembles the idea of Solidarity Lending which I first heard about when I read about Mohammad Yunus‘s Nobel Prize-winning work with the Grameen Bank.
Web3 applications take this a step further with “M-of-N” (multisig) transactions where the “N” is the pre-defined community of individuals that collectively decide on how transactions should be carried out.
Multisig transactions are also referred to as M-of-N transactions, with M being the required number of signatures or keys and N being the total number of signatures or keys involved in the transaction.
The idea of a collective making confirming a claim is a core principle of decentralized systems and it is encouraging to see it starting to emerge in mainstream applications. Could we ever make a sign-up/sign-in system based on collective claims as mainstream and the cryptic oAuth flows we have today with singular entities verifying claims? Here’s hoping 🤞 !
One of my most memorable/educational meetups was one by Philly.rb on audit logging. Audit logs record the occurrence of an event, the time at which it occurred, and any other useful data. The insight I remember from that talk was that audit logs are generated by creating a record for every change event in a new table. For example, your project may have a model User and a model ChangeOperation (usually generated by some library) and every time any data on the user model is changed, a new ChangeOperation instance is created and persisted in the database. You can later view all the ChangeOperation s the User model has gone through.
I am at EthDenver this week and it’s been great. There are some really interesting new projects using NFTs in a number of new ways. A typical NFT is such a simple idea that it is very amenable to representing different properties of a product. The Lens Protocol is one such interesting project that wants to represent social graphs on the blockchain. Like a few other social-network-on-the-chain projects, Lens uses NFTs to represent user identities.
Where Lens does something interesting though is representing verbs like “following” as an NFT. From their documentation:
Profile NFTs contain a FollowModule. This module contains the logic that allows different accounts to be issued Follow NFTs to record their relationship to the main profile on-chain.
Extracting the follow action to its own smart contract is an interesting move and allows you to create all sorts of various kinds of follows like “create a follow NFT only if the user has this other NFT” or “create a follow NFT if they pay a certain amount” (basically premium follows).
Definitely an interesting idea and something I need to dive into a bit more.
Code Zen 