Podcast with William Zeng Part 1
Summary
Whatif you had an AI-powered virtual assistant to handle all the tech debt and
coding tasks you dread? In this captivating podcast, we interview William Zeng,
the mastermind behind Sweep, an open-source AI-powered junior developer.
Explore the story behind Sweep's inception and how William's expertise in
statistics and machine learning led to revolutionary advancements in code
search and refactoring. Whether you're a seasoned developer or just starting,
this podcast will change the way you view coding forever.
Join us in this episode as we sit down with William Zeng, co-founder of Sweep and
former leader of Roblox's machine learning game search team. William's journey
from studying statistics and machine learning at Carnegie Mellon University to
developing AI-powered virtual assistants is nothing short of inspiring. Learn
how he drove double-digit growth in search CTR at Roblox and why he decided to
start Sweep. Discover how Sweep is transforming the way developers handle
technical debt and simplify code refactoring, making coding more efficient and
enjoyable.
[00:00:00] Andrew Liew: It's a great pleasure to inviteWilliam one of the co founder of sweep and let me share a bit. about William
and I'm going to read out a bit about him. So William studied statistics and
machine learning at Carnegie Mellon University, graduating in three years to
join Roblox. While at Roblox, he actually led the machine learning part of the
game search team, driving a cumulative double digit growth at the search CTR
over two years.
[00:00:33] Andrew Liew: He also trained and employed thefirst vector search model at Roblox. Eventually he is now working on Sweep, an
open source AI powered junior developer or so called like a robotic virtual
assistant. So you can actually check out Sweep by the end of this podcast. And
let us welcome William.
[00:00:51] Andrew Liew: William, thanks for coming to theshow.
[00:00:53] William Zeng: Yeah, Andrew super excited to behere and talk about Sweep and as well as introduce Sweep to the audience.
[00:00:59] Andrew Liew: [00:01:00]Yes. Before you do that, I think perhaps let the audience here and listen to
your backstory apart from me just sharing your profile. So tell us, like, how
did during school days or how do you, when you finish school, eventually went
to machine learning, eventually went to Roblox and discover this need to build
this tree?
[00:01:20] Andrew Liew: Of
[00:01:20] William Zeng: course. Yeah, I studiedstatistics at. And then there I learned the background of deep learning and how
to train these models. And then that kind of led me to roadblocks where I was
working on the search team. And I was really the, one of the first engineers to
work on search with an ML point of view.
[00:01:38] William Zeng: So then learning how to trainsearch models on a very. Different kind of data set, right? Roblox is very
unique. It has a lot of different types of games. So we need to handle that.
And I while I was there, I also saw a lot of tech debt. Like I saw a lot of
really talented engineers having to spend time doing tasks that I felt an AI
can handle.
[00:01:57] William Zeng: So that eventually with my cofounder, Kevin, we [00:02:00] decided to startsweep, which was a assistant to solve all of these, like the minor issues and.
Bugs that a human should not have to deal with. Okay, cool
[00:02:09] Andrew Liew: So let's talk a bit about thestory as you mentioned you while you're studying, statistics and machine
learning There's so many different classes of machine learning and application
like how do you eventually land yourself into you know the game search was it a
stumble upon or was it assigned to you or it was like The decision and oh, I'm
I decided to put my hands up.
[00:02:31] Andrew Liew: I want to go into this team
[00:02:32] William Zeng: Oh, of course. Yeah, so thenright after my my freshman year, I was applying for internships and then while
I was there, I was also taking, I actually had just changed more about my
background. I started as an architecture student in Carnegie Mellon. Yeah. And
then I was doing that and then I did that for about six weeks and then I got
really tired of it.
[00:02:55] William Zeng: I didn't really like drawing somuch. So then I explored a whole bunch of classes and [00:03:00]then I found out I really like statistics and kind of CS. So then I went more
towards that direction that went to this, major. Then after that summer, I took
this class called natural language processing, where I learned a lot about how
to the old ways of kind of training models to understand text and do actions.
[00:03:18] William Zeng: And then I eventually joinedwhile I was working there. It was just one games team and I was working there
and then eventually, and I did well over the summer. I came back as a full time
employee. I graduate early. I asked, I actually asked my manager at the time if
I could graduate early. And joined full time instead because I wanted to finish
school.
[00:03:37] William Zeng: It was right around the start ofCOVID. So I, wanted to be done with school, go to California and work here. So
then I came back and I joined the game search team. And from there I just kept
doing more projects.
[00:03:48] Andrew Liew: And okay, so while you're doingyour projects, how do you came upon this problem that you want to solve you
mentioned eventually you came up to sleep can you describe that story about
that, that discovering [00:04:00] moment or, itwas just like, so happened that, hey, this is something that, was it a pain, or
was it an itch, or what's that story like
[00:04:06] William Zeng: yeah, so then this kind ofstarted as I was working really hard on the search team.
[00:04:10] William Zeng: I really wanted this team to bereally great. When I went there, I thought, The team has had a lot to improve.
So I was really focused on that. And then at the same time, I'm like the first
hire or the first, the youngest person on the team. So I have to do a lot of
this tech debt, right? I have to clean up whatever is left over.
[00:04:27] William Zeng: Something comes in, I clean itup. And then I started thinking like this especially after last November OpenAI
released chat GPT. So I realized that with the combination of kind of. The
right context feeding the right files to the model. It can actually write
really good code.
[00:04:42] William Zeng: And some, pieces that I feltwere missing were like a really good search system. Not saying we have a the,
perfect one. We have a pretty good one at sweep. And then the other part was
the integrations, like directly making the PR instead of you have to go and
open your editor, because at that point it definitely helps open.
[00:04:59] William Zeng: I was doing a lot [00:05:00] of the heavy lifting here. But it stillrequires you to just open your ID, break your focus. And you still feel like
you're dealing with a problem. Whereas if you're reviewing the code, I didn't
feel so much. Like I was dealing with a problem more. I was like, Helping with
the problem, not I'm, the one being directly involved, right?
[00:05:15] William Zeng: That's how that
[00:05:16] Andrew Liew: idea came about. I see, becausebefore even this street came along I myself was a data scientist, I'm a
developer. So the typical old way of doing is literally you go in and you read
the stuff or Find a way to ask seniors about it.
[00:05:31] Andrew Liew: How do you read this messyspaghetti code thing? If the documentation is not there, I'm pretty much, you
have experienced that yourself and people will be thinking, Hey everybody faced
that, but how do you realize that? Hey, I can actually build this thing to
solve this problem. Every time this thing faces surfaces technical steps.
[00:05:50] William Zeng: Yeah, I think this reallystarted with I think this was last March myself My current co founder and one
of our, my, our friends, we [00:06:00] went tothis hackathon and we tried building this kind of coding agent that can build
an entire repository from scratch. And this is right around the time when also
other things were doing this as well auto GPT and, I kind of things. So we
built that and then we realized it's we, built that and we do all the hackathon
and we won a prize from from replic, which is one of these air companies. And
so then we did that and we realized this thing isn't actually that useful, but
it can write, you can write code, but it's not that useful code because most
people don't really need to create a website from scratch every once in a
while.
[00:06:35] William Zeng: And we thought there's aninteresting case, like if you're creating something from scratch, most likely
you can start with a boilerplate instead. If I want to create a website from
scratch, I can just find a template. And most of the time I can go and find if
it says blank blanks website, I can change it to William's website or Andrew's
website.
[00:06:52] William Zeng: I don't need to create the wholesite from scratch if it's not that useful. So then what we did was we thought,
okay, what's useful, like tech debt and like [00:07:00]working on really large code bases. So we really focus on handling that, that
kind of a problem.
[00:07:05] Andrew Liew: Ah, I see. Yeah. So you're right.Like, right now in terms of template, you have Wibbly, you have Wix, These are
pretty much a standard template website builder.
[00:07:16] Andrew Liew: Going into a large base of codeactually will require a large text of corpus because I used to have another
friend who actually try to solve this problem fixing code or reading
documentary code, interpreting in English and then helping the engineer make
sense of huge chunks of code especially when you have a lot of software
engineers working on it.
[00:07:37] Andrew Liew: And at a time before these largelanguage models came out, that was three, four years ago, it was very
painstaking you, have to pack the code structure, the code base, and then use.
Clustering and machine learning to make it with tokenization to enable some
level of interpretation.
[00:07:54] Andrew Liew: Like, what has changed? What doyou think has changed that, that enable you guys [00:08:00]actually getting more traction to solving these real problems? Yeah, I think
the,
[00:08:04] William Zeng: some of the things have changedare really powerful language models that can take any kind of adapt to new
problems. So before, if you take a kind of manual tagging solution.
[00:08:13] William Zeng: It, and then you don't reallyhave an engine that can make the logical changes to change the code, right? GPT
two, or wasn't going to do this. And even the kind of larger autoregressive
models are not going to do this. Whereas now you have something like GPT 3. 5,
we've actually been using deep in five for some of our tasks.
[00:08:32] William Zeng: It's still not that good. Ithink the really big catalyst that we found was when GPT four came out it was a
lot better. It's still not quite there yet, but it's much, much better than. 3.
5 at solving this problem. That was the first advancement. I think there still
is perhaps some more yet to be yet to come, probably improvements in language
models still need to happen.
[00:08:51] William Zeng: And, but the other one was ofcourse, really good search. I think there are really good tools right now.
Sourcecraft is an amazing code search engine, but I [00:09:00]think I think being able to take that and apply it to repos and an API way,
like how we were doing it, like just make a request and get back the files.
[00:09:09] William Zeng: With the smaller languagemodels, that is also a really big change that has helped. And you've seen this,
we've seen this a lot summarizing large PDFs. They're just doing a search over
the PDFs. And
[00:09:19] Andrew Liew: Because like you guys arefocusing on solving technical depth even solving technical depth, there's so
many angles that you can come from right?
[00:09:28] Andrew Liew: Where is that beachhead or theinitial preamble or primer? Technical use cases that you guys focus on that you
have some kind of tribe or community to say, Hey William and Kevin this is the
kind of problem set that we want to solve on technical gap.
[00:09:44] William Zeng: Yeah, so we're really focusedon, of course, listening to our users and seeing what they do and learning from
them.
[00:09:51] William Zeng: So the really big ones have beenadding, for example, adding logs and. Fixing front end changes have been really
good. I'll show a demo of that later. [00:10:00]And then some other ones that have been really good are refactors in one or two
files. So let's say you have one method that you want to move somewhere else
and maybe you want to also refactor it a little bit.
[00:10:10] William Zeng: Simplify the logic. That one is,it's really good at. So you can just do this. This is definitely just this is
still incredibly early for a sweep. So there's so much in the future doing this
iteratively over your entire code base, if you can go through and clean it up
and not just do this kind of without the human being involved, you really want
people to be involved so that someone can still understand and use it after you
can't just have a refactor your whole code base.
[00:10:34] William Zeng: It'd be really bad.