Disclaimer: I’m not a lawyer. This is not legal advice.

Benefits

There are generally two main benefits to forming an LLC:

1. Liability protection. You are not personally liable for the debts of the business.
2. Tax benefits. You can deduct qualified business expenses from the total income of the business. Here’s a simple example: If you spent $1,000 on servers and your website made $10,000, then you’ll only pay taxes on the $9,000 in profit.

How to form the LLC

There are three steps for forming an limited liability company (LLC) in California.

Step 0 - Choose a company name

Make sure that the company name you’d like to use is available for use.

• Use the California Business Search site to see if the name you want is available.
• Optional: Use US Trademark Search Database to see if your company name has been trademarked. You may run into trouble if someone has protected the name in the category of business that you’re starting. Also, if you decide to trademark the name in the future, then you’ll want to know it’s still available.

Step 1 – File Articles of Organization

To establish your LLC in California, you need to file Articles of Organization with the California Secretary of State.

• File Articles of Organization (Form LLC-1).
• Cost: $70, one-time

Wait for LLC-1 to be returned, which confirms your LLC is active, before completing the following steps.

Step 2 - File Statement of Information

The next step is to file a Statement of Information with the California Secretary of State. This lets California know some basic information about who is operating the LLC.

• File Statement of Information (Form LLC-12). This step can be completed online here.
• If your contact info has not changed since the last filing of LLC-12, file Statement of Information No Change (Form LLC-12NC) to save time. This step can be completed online here.
• Cost: $20, repeat every 2 years
• Complete this step within 90 days of filing LLC-1

Certain tasks must be completed on an ongoing basis to keep your company in good standing. Filing a statement of information is one such task.

• RECURRING: Add this to your calendar to repeat every 2 years: Re-file the Statement of Information

Step 3 - Pay annual tax

The next step is to pay taxes. Every LLC in California has to pay an “LLC Fee” of $800 per year, even if the LLC has no profits. Since you now have an active LLC, you must pay the current year’s LLC Fee even though it may be part-way through the year. The exact due date is technically the 15th day of the 4th month after the founding of the LLC.

• File Annual Tax (Form 3522). This step can be completed online here (select “Web Pay (Business)” and then “Annual Tax (Form 3522)”).
• Cost: $800, repeat every year
• Complete this step within approx. 4 months of filing LLC-1

In future years, the due date is April 15. For example: the 2019 LLC Fee is due on April 15, 2019. (Note: This is different than peronal taxes which are due the following year, i.e. for personal taxes, the 2019 taxes are due on April 15, 2020. But for an LLC the 2019 taxes are due on April 15, 2019)

• RECURRING: Add this to your calendar to repeat every year: File Annual Tax Form 3522

That’s it – you have an LLC! 🎉🎉🎉

Congrats, you have an LLC!

Generally, it’s also a really good idea to do the following recommended steps:

Step 4 - Get a Federal EIN number

Apply for a Federal EIN number online. An EIN number is like a social security number (SSN) for your company. It’s most useful for opening a bank account. You should use the EIN instead of your SSN whenever possible.

• Apply for a Federal EIN number online
• Cost: FREE, one-time

Step 5 - Open a business checking account

Open a business checkout account. Use your new EIN number instead of your SSN to open the account. Going forward, you’re going to want to use your business checking account (or an attached credit account) to pay for all business expenses.

Step 6 - Capitalize the business

Capitalize the business. That means putting some initial capital into the business. Transfer $1,000 - $5,000 of your own money into the business bank account to get the business started. This makes the business “real” and gives it some funds to start using to pay for expenses.

Step 7 - Create a Capital Contribution Agreement

Draft and sign a “Capital Contribution” to document which personal assets you want to transfer into the business. It’s important to document this so it’s very clear in case there’s ever a question about which assets are yours vs. the business’s.

Here’s a simple Captial Contribution template you can use:

INSERT_COMPANY_NAME – Capital Contribution (tax-free)

As of INSERT_DATE, all “INSERT_COMPANY_NAME assets” are transferred from INSERT_INDIVIDUAL_NAME (current owner) to INSERT_COMPANY_NAME (new owner).

“INSERT_COMPANY_NAME assets” include domain names and websites, code and servers, online accounts (GitHub, Twitter, etc.), business relationships, legal agreements, and intellectual property (copyrights, trademark rights, etc.).

Signed,

Current owner:
INSERT_INDIVIDUAL_NAME (sole proprietor)

---

New owner:
INSERT_COMPANY_NAME

---

INSERT_INDIVIDUAL_NAME (Member, acting on behalf of INSERT_COMPANY_NAME)

Step 8 - Keep personal and business expenses separate

Be diligent about keeping personal vs. business expenses separate. Or else, someone could claim your business is not really separate from you personally and attempt to sue you personally for the debts of your business. This is known as “piercing the corporate veil”.

Limited liability protection is one of the most important reasons for starting an LLC, so you don’t want to lose this protection by careless bookkeeping.

Step 9 - File a single tax return

When filing your personal taxes at the end of the year, your LLC’s income and expenses can be included with your own personal income/deductions. This is because a “single-member LLC” (an LLC owned by a single individual) is considered a “disregarded entity” for tax purposes. That means that it doesn’t have the obligation to file taxes separately. The tax obligations just pass straight through the LLC to you. This is nice, because it keeps things simple.

Nice work!

Hope this guide was helpful!

That’s why I’m super excited that libtorrent – the engine that powers many of the most popular torrent clients including qBittorrent, Deluge, and many more – has added support for the WebTorrent protocol.

WebTorrent support in libtorrent opens the door for many more torrent clients to connect to browser peers. Browser peers (which must use WebRTC) will now be able to access a huge trove of torrents currently only available to TCP/UDP peers.

The WebTorrent protocol allows peers to connect over WebRTC in addition to the widely supported TCP and UDP transports. In fact, UDP support itself was added to the BitTorrent protocol in a protocol extension (see the μTP protocol) and now UDP is the primary transport used by BitTorrent clients.

With this big news, we’re one step closer to the vision of browser-based torrents. One day soon, you’ll be able to navigate your web browser to any site with a JavaScript torrent implmentation embedded – like Instant.io or βTorrent – and be able to torrent anything available in the normal torrent network.

While desktop torrent clients aren’t going anywhere anytime soon, now the web browser will become a viable alternative to an installed torrent client. This is huge for less-technical users, users who can’t install native apps, or users who just feel safer using a website. WebTorrent offers more options and more ways to connect.

Torrent clients that can speak to both traditional TCP/UDP peers (orange) as well as the WebRTC-only browser peers (blue) are called “hybrid” peers (green). The libtorrent support for WebTorrent means that there are about to be a lot more hybrid peers!

The wider WebTorrent world

WebTorrent is more than a protocol extension to BitTorrent.

We build a popular desktop torrent client, WebTorrent Desktop, which supports powerful features like instant video streaming.

We also build a webtorrent JavaScript package which implements the full BitTorrent/WebTorrent protocol in JavaScript, the language of the web. This implementation uses TCP, UDP, and/or WebRTC for peer-to-peer transport in any environment – whether Node.js (TCP/UDP), Electron (TCP/UDP/WebRTC), or the web browser (WebRTC). In the browser, the webtorrent package uses WebRTC which doesn’t require a browser plugin, extension, or any kind of installation to work.

If you’re building a website and want to fetch files from a torrent, you can use webtorrent to do that directly client-side, in a decentralized manner. Our recently released WebTorrent Workshop is helpful for getting started and teaches you how to download and stream a torrent into an HTML page in just 10 lines of code.

const client = new WebTorrent()

const torrentId = 'https://webtorrent.io/torrents/sintel.torrent'
const torrent = client.add(torrentId)

torrent.on('ready', () => {
  // Torrents can contain many files. Let's use the .mp4 file
  const file = torrent.files.find(file => file.name.endsWith('.mp4'))

  // Display the file by adding it to the DOM
  file.appendTo('body', { autoplay: true, muted: true })
})

See this code in action on CodePen.

Also, I want to remind everyone that WebTorrent has been built into the popular, privacy-focused Brave browser since 2016. Just click on .torrent files or magnet links and they’ll magically work. And that’s all powered by WebTorrent.

With support for the WebTorrent protocol in libtorrent-based torrent clients, you’ll soon be able to connect to more peers which means faster, more reliable downloads.

The future is bright for WebTorrent!

The idea was this: whenever users install open source software, the funding package would display a message from a company that supports open source. The sponsorship would pay directly for maintainer time. That is, writing new features, fixing bugs, answering user questions, and improving documentation.

I appreciate the thoughtful discussion and feedback from the community. I’d like to share some thoughts about how the experiment went from my perspective.

The problem

Open source maintainers need income to fund their work.

Where did this start?

My objective was to try a new way to fund open source beyond just asking for donations. I’ve spent over 3,000 unpaid hours over the last four years maintaining some popular open source packages.

More broadly, I wanted to start a conversation about cultural norms around open source consumption.

Right now, the status quo is that maintainers create massive amounts of value and then for-profit companies and SaaS startups capture almost all of it.

As a case in point, let me tell you a story from the Open Source Summit conference I attended last week. While walking through the vendor area, I asked one of the startup reps “how much did you pay for this sponsor booth?” They weren’t sure if it was $10,000 or $20,000. This startup’s product is a package.json checker that tells companies if the open source software they’re using is permissively licensed (i.e. MIT, BSD, or Apache). They charge $50 per developer per month for this.

So this means that they charge a 50-person startup a whopping $30,000 per year to help them feel safe using code that open source authors like me have given away for free.

Meanwhile, the median startup likely contributes $0 per year to support open source maintainers.

I hold no grudge against this company – it seems like they’ve built a good product. But you have to admit, the fact that businesses will pay thousands of dollars for some SaaS software while ignoring the maintainers who write the actual open source code itself seems a bit unfair.

Maintainers do critical work which enables companies to create billions of dollars in value, yet we capture none of that value for ourselves.

Does it have to be like this?

I’m not arguing that maintainers should start capturing all of the value that we create. But we shouldn’t capture literally none of the value either. The status quo is not tenable.

I would love to find a way to help maintainers capture at least a bit of the value we create so that we can happily continue to write new features, fix bugs, answer user questions, improve documentation, and release innovative new software.

“Sustainability” only means subsistence

Increasingly, maintainers are starting to go through a mindset shift. We don’t want to ask politely for donations anymore – donations that often never come, or when they do they’re usually only enough for maintainers to sustain themselves but never enough to actually thrive.

Sustainability is another way to say subsistence. This is why the common phrase “open source sustainability” isn’t ideal.

I like what Kyle Mitchell wrote in Profit for Us, Sustainability for You:

“Profit” has a bad rep, in part because we don’t sully wages and fees for honest work by individuals with the term, even when they exceed immediate needs. It’s possible to profit without exploiting anyone. In fact, profit tends to show that you yourself aren’t being exploited. Your abilities, not your patron or your donors, should decide what kind and quality of life you can manage, and where. Value is leverage.

We should set our goals higher than subsistence. Folks who work on open source have a right to thrive and to be fairly compensated for our labor.

Remember: most open source maintainers can get a six-figure job relatively easily (in the SF bay area) where they’re just helping one company instead of helping the public. The opportunity cost is huge. We should be able to do what we want to help the public without making such a difficult choice.

Do we dare to hope for something better?

As long as significant personal sacrifice is a prerequisite for open source participation, we’ll continue to exclude a lot of smart and talented folks. This isn’t good for anyone.

And we’re forcing the folks who are able to participate to make extreme sacrifices that inevitably lead to burnout. Freeloading companies think they benefit from the current state of affairs. But maintainer burnout results in abandoned packages, ignored issues, unpatched security bugs, ecosystem churn, and more work for all of us. This is classic tragedy of the commons.

The dirty secret of open source is that much of it is powered by maintainer guilt.

As Dominic Tarr said after the event-stream compromise:

If it’s not fun anymore, you get literally nothing from maintaining a popular package.

But what if this weren’t the case?

What if anyone could make a living working on open source without needing to move to Thailand for lower cost-of-living, or needing to sell their startup and use the proceeds to work on open source full-time for four years, or needing to move to Hawaii to create a solar-powered farm and live frugally with near-zero income?

How much healthier and vibrant would the ecosystem be if less self-sacrifice was required? How many more people would be able to join in the fun and opportunity of open source?

What I’ve already tried

Starting in 2018, I began trying lots of different ways to raise funding so I could justify continuing to do open source full-time. The opportunity cost of forgoing a six-figure tech job are huge, especially when you live in the SF bay area.

Here are a few of the things I tried:

• thanks
• Patreon
• Tidelift
• GitHub Sponsors
• Paid consulting

With all these put together, I can make it work. But it still feels like a precarious solution. If even I – with my fortunate position – can’t make a comfortable living working on open source, then how is anyone else supposed to make it work?

Lots of maintainers struggle to reach a barely livable wage via donations. Of course, there are notable exceptions. But most maintainers are eventually forced to get a job making proprietary software, or at least split their time between open source and proprietary work which is far less beneficial to society. It’s unfortunate, but GitHub stars won’t pay the rent.

A lucky few manage to land day jobs that allow them to work on open source. But most folks have to be more creative – squeezing in time after work, secretly doing open source maintenance at work, or opting out of normal society completely. #solarpunk

I’m hopeful that GitHub Sponsors will make giving donations to maintainers more common. But I still worry a lot about folks who maintain packages that no one installs directly, i.e. transitive dependencies of popular packages. These maintainers have the hardest time.

The most common funding models – donations, README sponsors, or paid consulting – only work if a maintainer can get their appeal in front of users. This usually goes in a README or on a website.

But reliable, error-free transitive dependencies are invisible. Therefore, the maintainers are invisible, too. And, the better these maintainers do their job, the more invisible they are. No one ever visits a GitHub repository for a transitive dependency that works perfectly – there’s no reason to do so. But a developer investigating an error stack trace might visit the repository if for no other reason than to file an issue. At least then there’s a small chance they’ll see the maintainer’s plea in the README.

We need solutions that work for these folks too.

Maybe ads aren’t the answer – fine. But telling maintainers to bury their appeals where no one bothers to look is not the answer, either.

The story of funding

The seeds of an idea

When trying something new, you can’t plan for everything in advance. Sometimes the quickest way to learn if an idea is good or not is to just try it.

So, here was the rough idea for the funding experiment:

At a high level, an open source maintainer should be able to npm install funding into their project and start receiving money for their maintenance work.

Maintainers shouldn’t need to pitch fifty companies to find the rare one that is open to the idea of supporting open source financially. This is not work that most maintainers enjoy doing, are actually good at, or should need to do.

Rather than appeal to companies or users for donations – donations that often never come, and when they do, are never enough – funding lets maintainers take matters into their own hands. They can directly capture a small portion of the value that they’ve created. Not very much value, but at least not no value.

What if someone new to open source could adopt an abandoned package, quickly npm install funding, and start earning money for maintaining a package?

Suddenly formerly abandoned packages have a monetary value, and, therefore, a reason to be maintained. Bug fixes, security fixes, and pull requests would be less likely to be ignored. Folks who want to quit their day job to work full-time on open source now have a plausible means to do so. We all win.

What about when tons of sponsor messages start appearing in the terminal? How would this scale?

I had a plan for this. As more folks added funding to their packages, many copies of it might end up in the dependency tree. But funding de-duplicates the post-install message so it only ever prints one message.

As more maintainers join, funding ends up in more dependency trees so the single post-install message has more opportunities to be shown, bringing in more income for everyone. At some point, adding additional maintainers doesn’t help the message appear more times since it’s already showing in almost every install, so adding further maintainers only causes the total income to be split among more maintainers. We’d have to see how this played out. No one knows the value of a sponsor message in the terminal since this has never been tried before. That’s why this was an experiment.

Note that funding funnels funds to transitive dependencies much more effectively than any of the current funding approaches! That was an explicit design goal. There are lots of transitive dependencies that are “unpopular” in terms of mind share and GitHub stars, but which are used pervasively and yet invisibly. If this experiment worked, these “invisible maintainers” would have had a way to be rewarded for the value they provide.

Yuck. How would ads works?

Ads let us leverage the attention of the users of our software. They’re an imperfect solution because they extract value (in the form of a few seconds of attention) from all users equally – both from the open source “makers” as well as the “takers”.

For the record, funding had absolutely no tracking, no data collection, and no code from untrusted third parties. It was a console.log with some fancy formatting. Think of it like a newspaper classified ad. We just print it and hope that maybe some folks will see it.

But even when ads don’t track or collect data, they still feel “icky” to many users.

Oooo light bulb! Perhaps I should write a script that puts ads on the desktop wallpaper while installing ;) <evil laugh>

I'm all for sponsoring open source but ads during install are just annoying. Please don't. https://t.co/2knajtL0ay

— James M Snell (@jasnell) August 25, 2019

Personally, I’m not a huge fan of ads. I wish we could get away from ads as a funding model for free content. I use an ad blocker in my web browser, I install ad blockers on all my family’s computers, and I hope you do too. And yet, I’m pragmatic. I run ads on some of the websites that I’ve built and it helps fund ongoing maintenance work.

I supported anyone who wanted to block the sponsor messages from showing up. Which is why I added an environment variable OPEN_SOURCE_CONTRIBUTOR=true so that folks who support open source, whether through direct contributions or financially, could permanently silence the output. Obviously, even “takers” could use the flag, but hopefully the variable name makes them feel at least a little guilty.

The launch

I quietly launched this on August 19, 2019 without much fanfare. People who I know got in touch to express support.

The community reaction

Fellow open source maintainers and open source contributors have, by and large, been supportive of the experiment. Open source “consumers”, not so much.

Paolo Fragomeni said it best:

No one cool was upset by what @StandardJS did.

— Paolo F (@heapwolf) August 27, 2019

Folks who have spent time maintaining a popular open source project know how much time it takes to keep a project “healthy”. There are a constant stream of bugs to fix, new features to consider adding, pull requests to review, security issues in dependencies to resolve, user questions to answer, and underlying platform changes to deal with. Even for simple single-purpose packages, there’s a non-trivial ongoing maintenance burden. Especially when you’re maintaining hundreds of packages, as many in the Node.js community do.

good

— substack (@substack) August 25, 2019

Lots of maintainers welcomed the experiment as a much-needed conversation starter. That’s not to say that folks were thrilled with the idea of ads in their terminal. Many supporters expressed their dislike of advertising – even advertising where the sponsors are carefully selected and the implementation does not track users or collect data.

If nothing else, it’s nice that funding forced open source “consumers” – folks who enjoy the benefits of open source software without ever contributing anything back – to reconsider their relationship with open source. I think we successfully pushed back against the entitlement to free labor that is pervasive in the interactions that open source consumers have with maintainers.

I love your funding experiment @feross ❤️ don’t give up. Haters gonna hate.

— Jake Verbaten (@Raynos) August 25, 2019

@feross I'm glad you tried & ran the funding experiment, no matter what flak got thrown your way. :)

— noffle 🌱 (@noffle) August 27, 2019

The brigaders

Large numbers of the detractors seemed to come from the r/programming subreddit who are notoriously anti-JavaScript. A smaller number came from 4Chan and Hacker News. These drive-by condemners were eager to join in a pile-on in the standard issue tracker. But since these folks were neither users nor contributors to standard, I think their opinions should be discounted compared to those of actual users, fellow contributors, and fellow maintainers.

glob bless @feross for poking the hornet’s nest tho

— Forrest L Norvell (@othiym23) August 25, 2019

Folks who contribute nothing don’t get a seat at the table.

Rich Hickey said it best in Open Source is Not About You:

The only people entitled to say how open source ‘ought’ to work are people who run projects, and the scope of their entitlement extends only to their own projects.

As a user of something open source you are not … entitled to anything at all. You are not entitled to contribute. You are not entitled to features. You are not entitled to the attention of others. You are not entitled to having value attached to your complaints. You are not entitled to this explanation.

Ending the experiment

A new idea is delicate. It can be killed by a sneer or a yawn; it can be stabbed to death by a quip and worried to death by a frown on the right man’s brow. – Ovid

Since it seems clear this isn’t going to be the solution that saves us all, I’m ending the experiment. (In fact, it’s already been paused since Saturday when the initial two sponsors backed out.)

We reconsidered after reflecting on the developer community's reaction. We still passionately support open source software along with @feross, but we'll be more careful about experimenting in the future while continuing to innovate. On that note:https://t.co/WFrHBFGzul

— Linode (@linode) August 26, 2019

Both Linode and LogRocket are supportive of exploring new ways to support open source. It’s a bummer this didn’t work out, but I don’t hold it against either of them.

The future

I have plenty of interest from other sponsors and could keep working on funding with them onboard. One sponsor is particularly eager to start running their own terminal ad ASAP.

But I have other experiments in the works that I’m way more excited to try out.

And anyway, terminal ads seem like they have a limited lifetime. Specifically, there is an effort underway to replace post-install scripts with a field in package.json for soliciting donations. This will likely lead to the silencing of non-error install script output, which is honestly not a bad idea. The days of free-form post-install solicitations seem numbered. I expect that if npm makes this change they will couple it with automatic reporting of funding calls-to-action during npm install so as not to remove one of the few avenues maintainers currently have to reach users.

So while terminal ads may work for a little while longer, there are clearly better paths forward. Onward!

Approximately 100% of the Fortune 500 use open source code. Maintainers are just starting to wake up to our own power. Expect to be surprised. This certainly won’t be the last open source funding experiment.

Despite all the personal attacks against me, I’m really glad I ran this experiment. I think it was worthwhile because it seems to have moved the conversation forward, at least among the more thoughtful folks in the wider community.

I really appreciate the support that folks have shown for experimentation, and all the useful discussions that have resulted! Also, a huge thank you for the words of encouragement and support in the midst of all the harassment. Thank you everyone 🥰 🥰 🥰

Update: If you want to read more thoughts about funding from someone who’s a much better writer than me, I recommend these posts from Kyle Mitchell:

• Process of Elimination
• But You Said I Could

I’m releasing an open source funding experiment today.

The current model of sustaining open source is not working. We desperately need more experimentation. This is one such experiment.

npm install funding

What is this?

This is an open source funding experiment! ✨

Whenever users install open source software, this package will display a message from a company that supports open source. The sponsorship pays directly for maintainer time. That is, writing new features, fixing bugs, answering user questions, and improving documentation.

The goal is to make sure that packages are well-maintained now and for the foreseeable future, with regular releases, improved reliability, and timely security patches. Healthy open source packages benefit users and maintainers alike.

What does this code do?

You can take a look! All the code is open source in this GitHub repository. Essentially, it calls console.log() on some text. There is no tracking or data collecting — and it will always stay this way. You can look at the code to verify – indeed, this is the beauty of open source!

Where is this experiment running?

This experiment is currently running on a few open source projects that Feross maintains:

• standard

Update: The experiment is over. I wrote a recap post about it.

What is the long-term goal?

My goal with this experiment is to make StandardJS healthier. If we learn that the experiment works, perhaps we can help make all open source healthier, too. For complex reasons, companies are generally hesitant or unwilling to fund open source directly. When it does happen, it’s never enough and it never reaches packages which are transitive dependencies (i.e. packages that no one installs explicitly and therefore no one knows exists). Essentially, we have a public good which is consumed by huge numbers of users, but which almost no one pays for. Fortunately, there exists a funding model that usually works for public goods like this – ads. The goal of this experiment is to answer the question: Can we use ethical ads – ads that don’t track users or collect data – to fund open source software?

What will the funds be used for?

The funds raised so far ($2,000) have paid for Feross’s time to release Standard 14 which has taken around five days. If we are able to raise additional funds, the next thing we’d like to focus on is out-of-the-box TypeScript support in StandardJS (one of the most common feature requests!) and modernizing the various text editor plugins (many of which are currently unmaintained).

Where can I provide feedback about this experiment?

You can open an issue. But please be kind. I’m a human with feelings. ❤️

How can I disable this?

Just to be super clear: This package does no tracking or data collecting — and it will always stay this way. It’s just a fancy console.log().

If you support open source through direct contributions, donations, or however else you see fit, you can permanently silence funding by adding an environment variable OPEN_SOURCE_CONTRIBUTOR=true to your terminal environment.

Note, funding also respects npm’s loglevel setting, so e.g. npm install --silent and npm install --quiet will be respected.

I built a new web app – BitMidi – for listening to free MIDI songs. It’s a historical archive of MIDI files from the early web era. You should check it out at bitmidi.com.

Why build a MIDI website?

I was reminiscing about the days of Geocities and Angelfire, when the web was quirky and fun. In those days, sites often used the <bgsound> HTML tag to include an auto-playing song in the background of a page. Usually the song was in the MIDI file format.

MIDI files were the obvious choice for a few reasons.

First, MIDI files were one of the only audio formats supported by web browsers at the time, along with the WAVE and AU formats.

Second, they were relatively small compared to the contemporary alternatives like WAVE, AU, or even MP3. A typical MIDI file is usually less than 40 kilobytes, compared to multiple megabytes for WAVE, AU, or MP3 files. The secret of MIDI files’ incredibly small sizes is explained below!

It’s hard to describe MIDI files if you’ve never heard them before. The MIDI aesthetic is synth-laced, nostalgia-inducing, old-school, and endearing. It’s mixed up with the first DOS video games I ever played and the first websites that I visited as a small child in ‘95 and ‘96. A textual description can’t adequately capture the feeling of listening to a MIDI.

I was surprised and sad to learn that all major web browsers have removed MIDI file support. Perhaps this happened at the same time that the <bgsound> HTML tag was removed? This explains why I haven’t heard a MIDI in many years!

My quest to play a MIDI file

So, I really wanted to play a MIDI file. When I searching the web for a MIDI websites, I discovered that existing websites were quite disappointing. It was difficult to find a site with a decent user experience. Almost all the top sites were unmaintained or broken in various ways. Most sites could not play back the files directly in the site, requiring them to be downloaded and played in an external program instead.

Quicktime also dropped support for MIDI when Quicktime X was released in 2009 which inconvenienced many musicians and probably stopped many users from playing back (or discovering!) this wonderful file format.

Technically, Apple still offers Quicktime 7 for download, which can play MIDI. But it’s a 32-bit app so it will likely stop working once the macOS version after 10.14 Mojave is released. Also, embarrassingly, the Quicktime 7 download link uses insecure HTTP!

Even the venerable VLC failed to play back the MIDI files I threw at it. (I later learned that VLC can actually play MIDIs and I just got unlucky with the few I tried, hah!)

There were some websites that could convert MIDI to low-quality MP3, and some sites offered inline MIDI playback using Adobe Flash, but they were flaky and impossible to get working in Chrome.

Other websites implemented MIDI players in JavaScript, but they were usually missing instruments or extremely ineffecient multi-megabyte cross-compiled Emscripten libraries.

There was only one MIDI player website that really impressed me and that is Muki. Muki had the best-sounding web MIDI player but sadly their MIDI library, while impressive, is hard to browse since there’s no index listing, just radio mode and limited search. I wanted to lookup specific MIDIs that I remembered, like Age of Empires II Age of Kings.mid or aladdin2.mid or Curse of Monkey Island - The Barbery Coast.mid.

I eventually found a .ZIP archive with around 100,000 MIDI files. I wanted a way to search through them and share the best ones with friends. So I decided to build a site to address the shortcomings of existing MIDI sites.

And that’s BitMidi. A free way to browse and play MIDI files! ✨

If you’re curious how I built the site, there’s a section about that below. Keep reading.

Where did MIDI come from?

MIDI is an acronym that stands for “Musical Instrument Digital Interface”. It’s a way to connect devices that make or control sound — synthesizers, samplers, or computers — so they can communicate with each other, using MIDI messages.

The first synthesizer to support the MIDI standard, the Prophet-600, was released in December 1982. That’s over 35 years ago!

The MIDI protocol is an old protocol even compared to other important protocols. For example, HTTP is newer than MIDI!

• 1971 – FTP
• 1974 – TCP
• 1982 – MIDI
• 1991 – HTTP
• 1995 – SSH
• 2001 – BitTorrent
• 2009 – Bitcoin
• 2015 – HTTP2

How does MIDI work?

Sound is never sent via MIDI, just digital signals known as event messages. These messages instruct pieces of equipment (or software) to start playing sound. So you need an instrument (or a set of them) to actually produce sound from MIDI event messages!

The MIDI protocol supports 128 notes (around 10 octaves), 16 channels (so 16 separate devices can be controlled) and 128 programs (which are known as “patches” or instrument voices).

How is BitMidi built?

First step, I needed to build a MIDI player in JavaScript. At first, I was determinted to write one from scratch in JavaScript and use the Web Audio API to synthesize all the instruments in code. I thought this would yield the the smallest possible JavaScript file size.

However, I didn’t really have the audio engineering skills to pull this off. So ended up settling for an approach that uses SoundFonts, which are the same as the “patches” or instrument voices described above – basically just files that contain all the possible notes an instrument can play.

BitMidi uses the instrument voices from the General MIDI sound set released by FreePats.

Then I compiled the best MIDI player written in C (libtimidity) to WebAssembly using Emscripten. I put in lots of effort to optimize the built size and include the minimal amount of code. The result of my efforts are available in the npm package timidity. It’s quite lightweight – just 34 KB of JavaScript and 23 KB of lazy-loaded WebAssembly, smaller than anything I’ve seen on any other site.

Then I put a frontend on it, so it’s easy to browse all the files. BitMidi uses all the best techniques that I know about to make it super fast and snappy. The site gets perfect 100s on all categories on Chrome’s Lighthouse Performance benchmark, which is extremely non-trivial in my experience.

I’m slightly embarrassed to admit how proud I am of this:

I’m using Preact (a small 3kb alternative to React) and a small smattering of lightweight custom code to make the site super performant. The whole site is well below 50KB of JS, which is the best way to keep a site fast.

I plan to ingest a lot more MIDI files in the future, from sources like the Geocities MIDI archive on the Internet Archive and elsewhere.

Want a new MIDI every day?

I wrote a Twitter bot at @BitMidi that tweets out a new MIDI file every day. If that sounds awesome to you, then you should follow BitMidi on Twitter.

What are some fun or interesting MIDIs?

Here are a few of my favorites:

• Backstreet Boys - I Want It That Way MIDI
• Beethoven Moonlight Sonata MIDI
• Camptown Races MIDI
• Golden Sun - Overworld MIDI
• Kingdom Hearts - Dearly Beloved MIDI
• Legend of Zelda - Overworld MIDI
• Michael Jackson - Billie Jean MIDI
• Michael Jackson - Don’t Stop Till You Get Enough MIDI
• Passenger - Let Her Go MIDI
• Pokemon - Pokemon Center Theme MIDI
• Pokemon Red Blue Yellow - Opening MIDI
• Pokemon Red Blue Yellow - Wild Pokemon Battle MIDI
• Red Hot Chili Peppers - Californication MIDI
• Red Hot Chili Peppers - Otherside MIDI

Thanks for reading!

Feedback welcome! If you have thoughts on how to improve the site, please reach out and let me know what you think can be improved.

And please share this post and the BitMidi site with your friends that will enjoy it. ❤️ I’m hoping that you find it useful and entertaining.

🙌 I just released a neat little tool you can use with npm (the package manager for Node.js). It’s called thanks and it helps you donate to the open source maintainers you depend on! ✨

Here’s how to use it:

1. Run npx thanks in your Node.js project
2. See which of your dependencies are seeking donations! 💸

🌟 If you’re an open source author, add yourself to the list so people will know you’re seeking donations for your open source work:

I’m really excited by the opportunity to solve the crisis of “open source sustainability”. If you haven’t heard here’s the short version:

Our modern society—everything from hospitals to stock markets to newspapers to social media—runs on software. But take a closer look, and you’ll find that the tools we use to build software are buckling under demand.”

– Nadia Eghbal (“Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure”)

Lots of the open source that is used to build the top apps is actually in dire need of help. We need more people to maintain code and triage issues. But also more funding so we can justify spending more time on open source instead of working for clients/working at traditional jobs. I know maintainers who even need funding to feed/shelter themselves (this is not an exaggeration).

We have a lot more work to do. thanks is just one attempt to help, but it’s only the beginning.

For now, take a look and see if any of the software you depend has authors who are seeking donations to continue their work.

Happy hacking!

Update 2018-02-14: Post-mortem

• Built in 2-3 days
• Live streamed myself working on it on Twitch.

Response

• 1,100 retweets, 2,500 likes on the announcement tweet
• 1,100 GitHub stars
• #3 trending repo on GitHub for the day (#2 in JavaScript)
• Made it to homepage of Hacker News
• Made it to homepage of Product Hunt (#2 product of the day)

Forks

A few folks made versions for other languages:

• Python
• Ruby
• Go

Thoughts

Overall, much better response that I expected. In the future, I should try building out more ideas like this one where (1) I have a good gut feeling that there’s an unmet need, (2) it’s timely (current event or lots of ppl on twitter talking about it) and (3) it’s pretty quick to build. These are the most fun types of projects to work on :)

The idea that you can “always use semicolons” and not worry about Automatic Semicolon Insertion (ASI) is completely incorrect.

All JavaScript developers absolutely must understand ASI, even those who “always use semicolons”. Take this example:

function foo () {
  return
    {
      bar: 1,
      baz: 2
    };
}

Woops, you remembered to put a semicolon, but doesn’t matter. ASI kicked in and changed your code to:

function foo () {
  return; // <-- ASI adds a semicolon here. You now have a bug!
    {
      bar: 1,
      baz: 2
    };
}

So, it’s misleading to tell people that if they just “always use semicolons” their code is safe from surprising ASI behavior.

ASI will be with us forever. It’s about time you learned how it works. Not to worry: ASI is fully-specified in the ECMAScript language standard and all browsers implement it exactly the same way.

At the very least, consider using a linter that checks for unexpected ASI behavior. ESLint has a rule called no-unexpected-multiline which catches unexpected ASI behavior. And once you’re using a linter, it doesn’t matter whether you use or omit semicolons since the linter keeps you safe.

The argument for “never use semicolons”

It’s not actually that simple to “always use semicolons”. There are actually many edge cases where you still aren’t supposed to use a semicolon! For example:

function foo () {
  return 42; // ok
};           // <-- AVOID!

var foo = function () {
}; // ok

And what about these cases:

class Foo {
  constructor () {
    if (baz) {
      return 42; // ok
    };           // <-- AVOID!
    return 12;   // ok
  };             // <-- AVOID!
};               // <-- AVOID!

There are actually many more “edge cases” to keep in mind with “always use semicolons” than with “never use semicolons”.

If you “never use semicolons”, there’s only one rule: Never start a line with [, (, or `

In those cases, you simply prepend a ; like this:

;[1, 2, 3].forEach(bar)

However, if you frequently write code like this, you may be trying to be needlessly clever. This is actually much simpler:

const nums = [1, 2, 3]
nums.forEach(bar)

And if you use a linter like standard, then you don’t need to remember anything as unexpected ASI is reported as an error.

*The full list also includes some additional characters which would never actually appear at the start of an expression in real-world code: +, *, /, -, ,, .

Further reading

• JavaScript Standard Style
• An Open Letter to JavaScript Leaders Regarding Semicolons
• JavaScript Semicolon Insertion: Everything you need to know
• Are Semicolons Necessary in JavaScript? [YouTube]
• ESLint rule: no-unexpected-multiline

I’m Feross and I write open source software.

What is open source?

The term “open source” refers to code that other people can modify and share because the source code is publicly accessible.

Why is open source cool?

Open source software has brought enormous amounts of good to the world.

Open source is everywhere today. The majority of servers on the Internet are powered by Apache (an open source web server) and Linux (an open source operating system). Most networks in large corporations are powered by Linux, and even 95% of the desktops and servers at major Hollywood movie studios like Disney, Pixar, DreamWorks, and Sony run Linux.

Nearly everyone in the world directly benefits from open source software, whether they’ve heard of it or not.

How do I learn more about open source?

There’s an enormous trove of open source software on GitHub. You can learn a ton about coding just by reading code that’s been written by programmers in the past.

Try searching for some of your favorite projects, for example: jQuery, Node.js, Python, or Linux. You’ll find all the code that was written to make these projects work right there!

One good place to find some interesting projects is the Trending in Open Source page on GitHub.

How do I get involved in open source?

Whenever you find a bug or problem in one of these projects, you can go to GitHub to “file an issue” which is how you let the developers know about the problem.

If you’re feeling really brave and want to take your involvement to the next level, you can try fixing issues that have already been filed by downloading the code to your computer, fixing the problem, and sending your changes back to the project for inclusion. This is called “sending a pull request”.

If you want to get involved but aren’t sure where to start, check out My First PR which highlights good issues for first-timers to tackle!

Why do I write open source software?

I work on projects like WebTorrent, Standard JS, and Study Notes which are entirely open source.

Not everything I’ve worked on over my career has been open source, but most of it has been. I have over 100 open source projects on GitHub and npm. I am able to create lots of value in the world when I don’t restrict how people can use my work. Publishing things under an open source license lets anyone use my code however they like.

For example, my business Study Notes is doing extremely well in terms of making money even though the code is available on GitHub.

It’s possible to make a profit and share your code with the world :)

What tools do I use?

Some of my favorite tools at the moment are:

• Node.js
• GitHub
• Electron
• Standard JS
• Browserify
• Sublime Text

Happy hacking!

Why I’m a good fit:

• I’m the author of several popular npm packages, including WebTorrent, standard, and simple-peer.
• I help maintain Browserify, and wrote the buffer package for it.
• I’m a pragmatist. I care a lot about getting things done.
• I’ve built companies with Node.js:
  • PeerCDN, sold to Yahoo in 2013.
  • Study Notes, gets 1-2 million page views per month. This is still a solo project.
• I frequently help at NodeSchool events and co-founded a Stanford chapter.
• I’m not affiliated with any company, so I’m free to be an independent voice on the board.
• I’m organizing a fun conference in the Arctic circle :)

Our little community has pioneered so many great ideas in software! Node is fast, lightweight, and elegant. npm is the best package manager ever created. We’ve created so many clever, beautiful, and amazing hacks! 220,000+ experiments and counting…

Some of my favorite innovations include:

• The small module philosophy – write modules that do one thing well.
• OPEN Open Source – freely share commit access with people who make significant contributions to a project.
• Semantic versioning – use version numbers that mean something!
• Being nice humans – we try hard to create inclusive communities, assume good will, and be good people.
• The “Why not?” attitude – “Why are we doing this in JavaScript?” “Because we can!”

I care deeply about Node.js remaining the same pioneering and welcoming community that I fell in love with 4 years ago. This is the first programming community where I’ve felt truly at home.

I’ll do my best to keep Node.js awesome. Thank you for your consideration.

Yours humbly,

Feross

UPDATE: I will be representing the community on the Node Foundation board for a one-year term! Thanks to those who supported me and all who voted!

Hey everyone. @ag_dubs and I will be representing the community on the Node.js board! I'm honored and humbled. Will do my best!

— Feross (@feross) January 31, 2016

hey @nodejs community, @feross and I are your Individual Membership Directors

✨😳✨=> https://t.co/bUoH5Fb5dY

— ashley williams (@ag_dubs) January 31, 2016

Behold my latest hackery: FillDisk.com. Fill up your hard disk with just a single click!

How to troll using HTML5 localStorage

The HTML5 Web Storage standard was developed to allow sites to store larger amounts of data (like 5-10 MB) than was previously allowed by cookies (like 4KB). localStorage is awesome because it’s supported in all modern browsers (Chrome, Firefox 3.5+, Safari 4+, IE 8+, etc.).

The standard anticipated that sites might abuse this feature and advised that browsers limit the total amount of storage space that each origin could use. Quoting from the spec:

User agents should limit the total amount of space allowed for storage areas.

The current limits are:

• 2.5 MB per origin in Google Chrome
• 5 MB per origin in Mozilla Firefox and Opera
• 10 MB per origin in Internet Explorer

However, what if we get clever and make lots of subdomains like 1.filldisk.com, 2.filldisk.com, 3.filldisk.com, and so on? Should each subdomain get 5MB of space? The standard says no. Quoting the spec, again:

User agents should guard against sites storing data under the origins other affiliated sites, e.g. storing up to the limit in a1.example.com, a2.example.com, a3.example.com, etc, circumventing the main example.com storage limit.

A mostly arbitrary limit of five megabytes per origin is recommended.

However, Chrome, Safari, and IE currently do not implement any such “affiliated site” storage limit. Thus, cleverly coded websites, like FillDisk.com, have effectively unlimited storage space on visitor’s computers.

Proof-of-concept demo

Of course, I had to make a nifty demo to show how this works. See FillDisk.com as a proof-of-concept. You can get the source code on GitHub.

Features:

• Fills up the user’s hard disk on Chrome, Safari (iOS and desktop), and IE.
• Fills up 1 GB every 16 seconds on my Macbook Pro Retina (with solid state drive)
• Tested with Chrome 25, Safari 6, IE 10.
• For 32-bit browsers, like Chrome, the entire browser may crash before the disk is filled.
• Does not work on Firefox, since Firefox’s implementation of localStorage is smarter.
• Includes a button to reclaim your disk space ;)

Let’s fix this

Here are the bug reports I filed with Google, Apple, Microsoft, and Opera:

• Chromium bug report
• Apple bug report (on an unoffical site, since Apple doesn’t acknowledge their bugs publicly. So lame.)
• How do I file a bug on IE? Their [bug report](http://connect.microsoft.com/IE) page is broken.

  Microsoft bug report (requires login)

• Opera bug report (bug ID: DSK-383073, it’s private) - fills to 75MB in my testing, which isn’t so bad.

You can help get this issue get fixed quickly (on Chrome at least) by visiting the bug report pages and chiming in (starring) so that Google prioritizes this issue.

Update (Mar 13, 2013)

In the news:

• Ars Technica says this “could become a new form of Rick Roll”. Let’s hope not. :)

• The Verge

• BBC.co.uk

Answers to some common questions I’ve been getting:

• How did you find this HTML5 weakness?

  To be clear, this is not an issue with HTML5, so calling it an “HTML5 weakness” is not accurate. It’s a bug in the way that most browsers (Chrome, Internet Explorer, and Safari) have implemented the HTML5 Web Storage standard. It’s the fault of the browsers, not the HTML5 spec.

  I usually don’t spend my time actively looking for security bugs – I just frequently run into them while programming. In the case of the localStorage bug, I found it while working on a project that utilizes localStorage. I was curious if there was some way to store more than 2.5-10 MB using the API, and using multiple site “origins” by creating tons of subdomains was the first thing that came to mind.

• You mention Firefox uses localStorage and that it’s better but is there a reason for this?

  Firefox simply places a 10 MB cap on the amount of space that any domain can store using localStorage. So, origins like 1.filldisk.com and 2.filldisk.com, etc. must all share 10MB of space. Making new subdomains doesn’t give you more space.

• Have you had any followup since leaving the bug reports for Google, Opera, Apple and Microsoft?

  903 people starred the Chromium bug report I created, making it the 9th most starred bug in all of Chromium. The Chrome devs have responded and are working on a fix, though it’s unclear when it will be ready. Apple has not responded, because, well, they’re Apple. Microsoft has responded that they “will be investigating this issue further”.

  So, it seems that the issue will be fixed at some point, though it doesn’t appear to be a top priority for browser vendors at the moment.

I visited Japan for two weeks with some friends. It was awesome! The food was delicious, the sights were exotic and beautiful, and the culture was fascinating and different.

Here’s a picture of me at the top of Mt. Fuji, the tallest mountain in Japan:

We started in Tokyo, and explored many different districts, each with its own unique vibe. We went to Shibuya (the entertainment district), Shinjuku (home to the busiest train station in the world), Asakusa (old Tokyo), Akihabara (the electronics and otaku district), Odaiba (futuristic architecture on a man-made island), and also visited the Hama Rikyu gardens, the Tokyo Tower, and it’s modern replacement the Tokyo Skytree.

After Tokyo, we climbed Mt. Fuji on a whim (the view at the top was so amazing – see pictures below). We also visited Hiroshima (the museum was excellent, but understandably a bit morose), and Kyoto where we visited several temples and shrines, as well as the International Manga Museum. Before we left Kyoto, we paddled a rowboat down the river which was surprisingly hard (we bumped into other river traffic more than once!), but lots of fun.

Lastly, we visited Rabbit Island, which is a small island near Hiroshima, that is overrun by tens of thousands of tame rabbits. It was quite a sight!

We arrived at dusk and there were at least ten times as many rabbits as during the daytime, since rabbits are crepuscular, meaning they are most active during the twilight hours.

Some things that make Japan unique

I wanted to share some of the obvious things that make Japan “different”:

• Everyone plans stuff in advance. The Japanese customs agents were surprised and incredulous that we showed up in Japan at 10 PM without a hotel booked in advance for our first night. Travelers and locals who we met along the way were also surprised that we were “winging it”. The extra flexibility was awesome, and we only had trouble finding a hotel on one particular night in Kyoto because it was a holiday.
• Old people are treated with much respect. When we were in Kyoto, it happened to be a holiday: “Respect for the Aged Day”.
• Everyone takes their shoes off inside their homes.
• Most hotels require you to leave your room key at the front desk when you go out (so you don’t lose it?)
• Most hotels have a curfew (usually 11 or midnight)
• People are REALLY quiet on trains.
• People usually don’t walk and eat at the same time. It’s considered rude.
• It’s illegal to smoke cigarettes outside on the sidewalk (to prevent cigarette butts from being everywhere on the street)
• People shower BEFORE taking a bath. People bathe together in groups.
• Most salarymen wear the same “uniform” (white collar shirt, black slacks)
• Most students wear the same uniform (white polo shirt, black slacks/skirt)
• Everyone was well-dressed. No one we saw in public looked like a bum – which is definitely not the case in the US.
• I didn’t see a single homeless person in any of the cities we traveled to.
• Everything tends to opens late (~11am) and close early (~5pm)
• Many fast food restaurants have machines where you buy a ticket which you can exchange for food. So you basically pay a machine.
• English is everywhere (lots on signs, subways, menus, and most people who work in hotels and restaurants know enough words that you can communicate with them).
• Trains / buses are always on time (except for one private bus at Mt. Fuji which was 10 min late)
• Anytime an elevator or door closes with an elevator attendant or conductor behind it, they do a small bow until the door is fully closed.
• A few hotels we stayed at did not allow guests with tattoos because traditionally only yakuza (Japanese mafia) have inked their bodies. Kids are starting to get tattoos as fashion statements in recent years, but most bathhouses and hotels still ban it.
• A pretty large number of people used umbrellas to protect their skin from the sun (probably like 10-20% of people who were outside, mostly women)

You probably noticed that lots of the items in this list are obvious, surface-level differences. In the future, I want to write a longer post about the subtler (and more important) things that make Japan unique.

For now, I’ll leave you with a huge pile of photos from our trip.

Photos from our travels!

You can use the arrow keys to navigate through the photos. There are descriptions on many of the photos with the “what” and “where”.

Last thing: If you’re planning a trip to Japan and have questions, feel free to send me an email and I’ll do my best to help you out.

Japan - Day 1

First day of our travels in Japan!

Japan - Day 2

Our second day in Japan. We visited Akihabara (the electronics district) in Tokyo. Then, we took a boat down the Sumida River to visit Odaiba (a modern commercial district with lots of american-style malls).

Japan - Day 3

Our third day in Japan! We visited Kamakura today (a very old city with dozens of shinto shrines and buddhist temples). We went to the biggest and most famous shrine in the city, Tsurugaoka Hachiman-gū, which was founded 1000 years ago. In the evening, we went to the beach and happened to meet some nice Japanese girls who spoke a little bit of English. Good day.

Japan - Day 4

Our fourth day in Japan! We visited an onsen (Japanese-style bath), the Tokyo Imperial Palace (briefly), Shibuya district (the entertainment, culture, and fashion district of Tokyo) (also briefly), and stayed the night in Shinjuku (a really busy city with lots of skyscrapers).

Japan - Days 5 and 6 [Climbing Mt. Fuji]

On our fifth day in Japan, we visited the Tsukiji Fish Market (the largest fish market in the world!) and decided to summit Mt. Fuji, even though we have like no climbing experience. We didn’t check the weather and didn’t really know if it would be feasible, but we figured we’d try nonetheless. After 7 hours of hiking and a short nap, we made it to the top of the mountain – 12,000+ feet high! The view from the top was amazing and totally worth the hike!

At the top of Mt. Fuji there was no running water or restrooms, but I could access Google, Facebook, email, and Spotify. I had perfect signal strength. This situation felt odd. :)

Japan - Days 7 and 8

Our Japan trip goes on! Today, we visited Fuji-Q, an awesome amusement park right next to Mt. Fuji. Then, we took the shinkansen (bullet train) to Kyoto, where we checked out some temples and walked the “Philosopher’s Trail”.

Japan - Day 9

Today, we visited Western Kyoto and Arashiyama. We walked around Tenryuji Temple and paddled a row boat down the river. So much fun! Then, we came back to Kyoto and just chilled in the city. I randomly met some Australian girls who were staying in the same hotel. They were high schoolers in Japan for a field trip (or as they put it, an “excursion”). It was nice to meet some English-speakers!

Japan - Days 10 and 11

Japan trip is almost over! :( Today we checked out more of Kyoto (Nishihongan-ji Temple, the Imperial Palace, the International Manga Museum, and Nijō Castle). Then, we went to Otsu Beach, since we felt like swimming but it turned out we were misled – the beach was pretty uninviting despite what the city’s tourism website said. But, it was still a fun little trip to a REALLY rural part of Japan. The next day, we toured Hiroshima, saw the Hiroshima Peace Memorial Park, checked out the museum (which was a little depressing, but also really interesting).

Japan - Day 12 [Rabbit Island!]

Today we visited Okunoshima, aka Rabbit Island! The island was used as a poison gas factory during World War II and they tested the chemicals on rabbits. After the war ended, they dismantled the factory and set the rabbits free on the island. They multiplied like crazy and now there are hordes of tame rabbits all over the island. Whoever set the first rabbits free is a genius. This was so much fun! Later, we rode the bullet train back to Tokyo and checked out Shibuya for a few hours before calling it a night.

Japan - Day 13

Our last day in Japan! We ate breakfast at the Tsukiji Fish Market (one of the largest in the world). We ate fresh sushi, best that we had the whole trip. Then, we wandered around Shibuya for the rest of the day. Jake went shopping for hipster clothes. It was clear that nothing would fit me, since their “large” is like a small or medium here. The salespeople mostly ignored me :( Then, we flew back to SFO. Overall, awesome trip. Way better than I expected. I need to travel more.

Quick! Click this link to Bank Of America. There’s nothing fishy about it at all! I promise!

Go ahead – hover your mouse over the link to see where it goes. You’ll find that it’s a completely normal link to https://www.bankofamerica.com.

There is only one way to find out if I’m telling the truth – just click the link already!

[NOTE: The demo only works with a normal click on the link. No “Open in New Tab” or middle-click.]

What just happened?

Ok, I lied – the link was pretty fishy afterall. When you click on the link, you don’t actually navigate to https://www.bankofamerica.com. Instead, your browser automatically enters fullscreen mode and I load a fake version of Bank of America’s website (my demo uses a screenshot, but attackers would use a working website).

The fake Bank of America site is adorned with OS and browser UI that indicates you are actually on https://www.bankofamerica.com. Of course, these UI components are just screenshots too! However, they’re pretty convincing because they actually take into account the OS and browser you are using!

Also, note the green lock in the location bar, which indicates that TLS (i.e. HTTPS) is enabled.

The “Fullscreen API” explained

The Fullscreen API (see W3C docs and MDN docs) allows web developers show web content that fills up the user’s screen completely. You’ve seen this functionality in action whenever you watch a fullscreen video on YouTube (if you use their new HTML5 player, which you should do!) or look at a fullscreen photo on Facebook.

Note that most browsers have had user-triggerable full-screen functionality for some time now. The HTML5 Fullscreen API is distinct from this; it allows the web developer to access this same functionality, and importantly, the developer can trigger it programmatically.

This is nice because the developer can design a fullscreen button which looks like part of their site (a la YouTube and Facbeook). You can trigger fullscreen mode with this code:

elementToMakeFullscreen.requestFullScreen();

The main restriction that the API places on developers is that fullscreen must be triggered in reaction to a click or keypress. Presumably, this is so that sketchy sites can’t immediately put you into fullscreen when you land on their site.

// Assuming jQuery is available

// Fullscreen the HTML document on click
$('#fullscreen-button').on('click', function() {
  var doc = document.documentElement;
  if (doc.requestFullscreen) {
    doc.requestFullscreen();
  }
});

Note that in practice, you need to use the prefixed versions (mozRequestFullScreen() and webkitRequestFullScreen()) since the spec is still not final yet.

How the attack works

Create a link to a site that the user trusts:

Visit <a href="https://www.bankofamerica.com">Bank of America</a> for mediocre banking services.

The user can hover their mouse over the link and their status bar will show https://www.bankofamerica.com, as expected.

However, when the user clicks the link, call event.preventDefault() to prevent the browser from actually navigating to the link. Instead, trigger fullscreen mode and insert fake OS and browser UI into the page, along with a fake version of the site to be phished.

$('html').on('click keypress', 'a', function(event) {

  // Prevent navigation to legit link
  event.preventDefault();
  event.stopPropagation();

  // Trigger fullscreen
  if (elementPrototype.requestFullscreen) {
    document.documentElement.requestFullscreen();
  } else if (elementPrototype.webkitRequestFullScreen) {
    document.documentElement.webkitRequestFullScreen(Element.ALLOW_KEYBOARD_INPUT);
  } else if (elementPrototype.mozRequestFullScreen) {
    document.documentElement.mozRequestFullScreen();
  } else {
    // fail silently
  }

  // Show fake OS and browser UI
  $('#menu, #browser').show();

  // Show fake target site
  $('#target-site').show();
});

It’s important that the fake OS and browser UI match the user’s system. So, if Chrome user on OS X clicks the link, we show a fake OS X menu bar and fake Chrome UI with a green padlock for HTTPS on Bank of America.

How I built this

I built a working demo, which I encourage you to check out. You activate it by simply clicking the link in the previous sentence. You’ll want to use Chrome, Firefox, or Safari for the demo to work.

To make the demo, I took screenshots of different OS and browser chromes (Chrome, Firefox, and Safari on each of Windows, OS X, and Ubuntu) and split the images so I could use them in a fluid layout (to handle aribitary screen resolutions). I make sure to preload all the screenshots in the background so that the UI will be ready when the user clicks the target link.

The demo also works if the user is already in fullscreen mode when they click the target link.

The demo’s source code is also available on GitHub.

Would this actually work in the wild?

It’s true that the OS and browser UI won’t be exactly perfect for every user’s system. Lots of people customize their OS and browser and some might notice that they’re looking at fake UI.

For example, the user’s browser bookmarks and running menu bar apps may be different from the screenshots. The clock time in the OS X menu bar will almost certainly be wrong. One could improve the demo by adding the correct time to the OS UI, but this really isn’t necessary.

On OS X, Chrome plays an annoying 1-second animation any time you go fullscreen, which might set off some alarm bells with experienced users.

Despite all these apparent shortcomings, this remains a very serious attack because of the phenomenon of change blindness.

[Change blindness] is the phenomenon where seemingly striking or obvious changes are not noticed. – Milan Verma on BBC

Check out this excellent video from the psychology study where “change blindness” was first proven outside the lab:

Read more about change blindness on Wikipedia.

Humans are terrible at spotting subtle changes

If this attack were used in the wild, I bet at least 10% of web users would get phished (probably many more).

Links are the bread and butter of the web. People click links all day long – people are pretty trained to think that clicking a link on the web is safe. Saavy users may check the link’s destination in the status bar before clicking, however, in this case it won’t do them any good.

Most browsers don’t do a good job of making it clear that the browser just entered fullscreen mode. Safari shows a quick half-second animation, then no indication at all that the browser is in fullscreen mode, making users susceptible to all kinds of phsishing attacks involving fake OS and browser UI.

Chrome and Firefox (especially recent versions) do a better job of putting their own UI on top of the fullscreened content, but in Chrome especially, it’s pretty subtle and easily missed.

I’m sure we can all think of friends or family that would be fooled by a trick like this. After all, enough people still respond to Nigerian scam emails that it’s profitible to continue sending them!

A quick note about “features vs. security” in software

In software engineering, functionality and security are at odds with each other. When you add a new feature to a complex system, it’s impossible to predict how the new feature will interact with each of the thousands of existing features, in all their myriad combinations.

When the fullscreen API was first drafted by Mozilla, they predicted attacks like this:

Browser vendors are well aware of the potential security issues with fullscreen. For example, a malicious site could show a full screen Windows or Mac login window and steal a password. That’s why they are disabling keyboard support by default and only enabling by explicitly asking. – John Dyer

At some point, Mozilla (as well as the other browser vendors) must have decided that keyboard support in fullscreen mode is useful enough to legit web developers that it’s worth taking a hit to security.

I’m not convinced that this was the right tradeoff to make. What do you think?

UPDATE (Oct 9, 2012, 12:00 AM): The Mozilla developer documentation says this:

Also, any alphanumeric keyboard input while in full-screen mode causes a warning message to appear; this is done to help guard against phishing attacks. The following keys are the only ones that don’t cause this warning message to appear (…) – MDN

However, this documentation is out-of-date. There were no warnings on keyboard input in Firefox or Chrome. I went fullscreen on a Facebook photo and was able to leave a comment without any issues. Safari, on the other hand, appears to prevent keyboard input.

UPDATE (Oct 9, 2012, 12:22 AM): Someone on Hacker News pointed out that Internet Explorer used to allow the creation of chromeless popup windows which could be augmented with fake OS or browser UI to create phishing attacks. This feature was removed in Internet Explorer in 2004.

UPDATE (Oct 18, 2012): Google, Mozilla, and Apple are discussing what (if anything) to do about the issue I demonstrated. I replied on the “public webapps” W3C mailing list with my thoughts.

Shoutout to Neal Wu!

So, you followed the advice in my Linode Hosting Review and decided to host your website with Linode. Excellent choice!

But, if you’re new to the command line (perhaps you are familiar with CPanel or Plesk?) or you’ve never setup a server from scratch before, you may be wondering what dark magic vudu is required to get up and running.

Well, you’ve come to the right place!

The Command Line Is Hard (…at first)

I’ve set up at least five new servers with Linode and each time I complete the ritual, I learn new incantations that make the Linux angels sing. I’m pretty happy with my current recipe.

Setting up a new server can be confusing, so using a tutorial like this one is a good idea the first time you do it.

Tutorial: How To Set Up Your Linode

In this guide, I will demonstrate how to set up a fresh Ubuntu server from scratch, update everything, install essential software, lock down the server to make it more resilient against basic attacks and denial-of-service, improve server stability, setup automatic backups to another server, and finally install common software like Nginx, MySQL, Python, Node, etc.

A Note About This Guide

I originally compiled this guide as a .txt file of notes for myself, but decided to share it in case anyone finds it useful. If you’re looking for something straight from the horse’s mouth, Linode also offers guides that cover how to set up a new server.

Let’s get started!

Provision a New Linode

First, you need to provision a new Linode. Using Linode’s web UI, it’s quite easy. Select your desired Linode size. If you’re unsure, choose the smallest size. You can always resize it later. You also need to select a location for the server. I usually select “Fremont, CA” since that is closest to my location and the location of most of my users (United States, west coast).

Next, let’s install an OS. Select the “Rebuild” tab. Pick “Ubuntu 18.04 LTS” for the OS image. Use 256MB as the swap disk size (it is default). You’ll be asked to create a password for the root user.

After a few minutes, your server will be ready. Now, click “Boot” to get things started!

Next, let’s connect to the server.

Connecting to Your Server

First, open Terminal on your Mac. On Windows, you’ll want to use putty or install the Windows Subsystem for Linux to get a proper Terminal.

To connect to your server, type this into your terminal and hit Enter:

ssh root@<your server ip>

Of course, replace <your server ip> with your Linode’s actual IP address, which you can find on the “Remote Access” tab in the control panel.

This command launches the SSH program and asks it to connect to your server with the username root, which is the default Ubuntu user. You will be prompted for the root password you created earlier.

Basic Ubuntu Setup

To set up your new server, execute the following commands.

Set the hostname

Set the server hostname. Any name will do — just make it memorable. In this example, I chose “future”.

echo "future" > /etc/hostname
hostname -F /etc/hostname

Let’s verify that it was set correctly:

hostname

Set the fully-qualified domain name

Set the FQDN of the server by editing the /etc/hosts file:

vim /etc/hosts

Make sure the following line is in the /etc/hosts file (after anything that’s in there by default):

<your server ip>   future.<your domain>.net       future

It is useful if you add an A record that points from some domain you control (in this case I used “future.<your domain>.net”) to your server IP address. This way, you can easily reference the IP address of your server when you SSH into it, like so:

ssh future.<your domain>.net

If you’re curious, you can read more about the /etc/hosts file.

Set the time

Set the server timezone:

dpkg-reconfigure tzdata

Verify that the date is correct:

date

Update the server

Check for updates and install:

apt update
apt upgrade

Basic Security Setup

Create a new user

The root user has a lot of power on your server. It has the power to read, write, and execute any file on the server. It’s not advisable to use root for day-to-day server tasks. For those tasks, use a user account with normal permissions.

Add a new user:

adduser <your username>

Add the user to the sudo group:

usermod -a -G sudo <your username>

This allows you to perform actions that require root privilege by simply prepending the word sudo to the command. You may need to type your password to confirm your intentions.

Login with new user:

exit
ssh <your username>@<your server ip>

Set up SSH keys

SSH keys allow you to login to your server without a password. For this reason, you’ll want to set this up on your primary computer (definitely not a public or shared computer!). SSH keys are very convenient and don’t make your server any less secure.

If you’ve already generated SSH keys before (maybe for your GitHub account?), then you can skip the next step.

Generate SSH keys

Generate SSH keys with the following command:

(NOTE: Be sure to run this on your local computer – not your server!)

ssh-keygen -t rsa -b 4096 -C "<your email address>"

When prompted, just accept the default locations for the keyfiles. Also, you’ll want to choose a nice, strong password for your key. If you’re on Mac, you can save the password in your keychain so you won’t have to type it in repeatedly.

Now you should have two keyfiles, one public and one private, in the ~/.ssh folder.

If you want more information about SSH keys, GitHub has a great guide.

Copy the public key to server

Now, copy your public key to the server. This tells the server that it should allow anyone with your private key to access the server. This is why we set a password on the private key earlier.

From your local machine, run:

scp ~/.ssh/id_rsa.pub <your username>@<your server ip>:

On your Linode, run:

mkdir .ssh
mv id_rsa.pub .ssh/authorized_keys
chown -R <your username>:<your username> .ssh
chmod 700 .ssh
chmod 600 .ssh/authorized_keys

Disable remote root login and change the SSH port

Since all Ubuntu servers have a root user and most servers run SSH on port 22 (the default), criminals often try to guess the root password using automated attacks that try many thousands of passwords in a very short time. This is a common attack that nearly all servers will face.

We can make things substantially more difficult for automated attackers by preventing the root user from logging in over SSH and changing our SSH port to something less obvious. This will prevent the vast majority of automatic attacks.

Disable remote root login and change SSH port:

sudo vim /etc/ssh/sshd_config

Set “Port” to “444” and “PermitRootLogin” to “no”. Save the file and restart the SSH service:

sudo systemctl restart ssh

In this example, we changed the port to 444. So, now to connect to the server, we need to run:

ssh <your username>@future.<your domain>.net -p 444

Update: Somone posted this useful thought about choosing an SSH port on Hacker News:

Make sure your SSH port is below 1024 (but still not 22). Reason being if your Linode is ever compromised a bad user may be able to crash sshd and run their own rogue sshd as a non root user since your original port is configured >1024. (More info here)

Advanced Security Setup

Prevent repeated login attempts with Fail2Ban

Fail2Ban is a security tool to prevent repeated failed login attempts from attackers. It works by monitoring important services (like SSH) and blocking IP addresses which appear to be malicious (i.e. they are failing too many login attempts because they are guessing passwords).

Install Fail2Ban:

sudo apt install fail2ban

Configure Fail2Ban:

Setup configuration in a new file (will overwrite defaults in /etc/fail2ban/jail.conf):

sudo vim /etc/fail2ban/jail.local

Paste the following into /etc/fail2ban/jail.local:

[DEFAULT]
destemail = <your email address>
sendername = Fail2Ban

[sshd]
enabled = true
port = 444

[sshd-ddos]
enabled = true
port = 444

(Change the port number to match whatever you used as your SSH port).

Save the file and restart Fail2Ban to put the new rules into effect:

sudo systemctl restart fail2ban

Add a firewall

We’ll add an iptables firewall to the server that blocks all incoming and outgoing connections except for ones that we manually approve. This way, only the services we choose can communicate with the internet.

The firewall has no rules yet. Check it out:

sudo iptables -L
sudo ip6tables -L

Next, we’ll install a package which enables persistent firewall rules. This means that the firewall rules will get automatically applied at server startup:

sudo apt install iptables-persistent

When prompted, agree to have the current rules installed into /etc/iptables/rules.v4 and /etc/iptables/rules.v6.

Setup the IPv4 firewall rules in /etc/iptables/rules.v4:

sudo vim /etc/iptables/rules.v4

The following firewall rules will allow HTTP (80), HTTPS (443), SSH (444 (our custom SSH port)), ping, and some ports for testing. All other ports will be blocked.

Paste the following into /etc/iptables/rules.v4:

*filter

# Accept established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow loopback interface traffic
-A INPUT -i lo -j ACCEPT

# Reject non-loopback interface traffic to loopback IP addresses
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT

# Allow outbound traffic
-A OUTPUT -j ACCEPT

# Allow HTTP and HTTPS traffic
-A INPUT -p tcp -m state --state NEW --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 443 -j ACCEPT

# Allow SSH connections (Note: --dport should match the port in sshd_config)
-A INPUT -p tcp -m state --state NEW --dport 444 -j ACCEPT

# Allow ping
-A INPUT -p icmp -m state --state NEW -m icmp --icmp-type 8 -j ACCEPT

# Allow test ports
-A INPUT -p tcp -m state --state NEW --dport 8080:8090 -j ACCEPT

# Log denied connections
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

# Reject all other inbound and forward traffic (default deny unless explicitly allowed)
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

Setup the IPv6 firewall rules in /etc/iptables/rules.v6:

sudo vim /etc/iptables/rules.v6

Paste the following into /etc/iptables/rules.v6:

*filter

# Accept established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow loopback interface traffic
-A INPUT -i lo -j ACCEPT

# Reject non-loopback interface traffic to loopback IP addresses
-A INPUT ! -i lo -d ::1/128 -j REJECT

# Allow outbound traffic
-A OUTPUT -j ACCEPT

# Allow ping
-A INPUT -p icmpv6 -m state --state NEW -m icmpv6 --icmpv6-type 8 -j ACCEPT

# Log denied connections
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

# Reject all other inbound and forward traffic (default deny unless explicitly allowed)
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

Activate the firewall rules now:

sudo iptables-restore < /etc/iptables/rules.v4
sudo ip6tables-restore < /etc/iptables/rules.v6

Verify that the rules were installed correctly:

sudo iptables -L
sudo ip6tables -L

Restart the server and confirm that the rules are still in place.

sudo shutdown -r -h +0
ssh <your username>@future.<your domain>.net -p 444
sudo iptables -L
sudo ip6tables -L

Get an email anytime a user uses sudo

I like to get an email anytime someone uses sudo. This way, I have a “paper trail” of sorts, in case anything bad happens to my server. I use a Gmail filter to file these away and only look at them occasionally.

Create a new file for the sudo settings:

sudo vim /etc/sudoers.d/my_sudoers

Add this to the file:

Defaults    mail_always
Defaults    mailto="<your email address>"

Set permissions on the file:

sudo chmod 0440 /etc/sudoers.d/my_sudoers

This is isn’t mentioned anywhere on the web, as far as I know, but in order for the “mail on sudo use” feature to work, you need to install an MTA server. sendmail is a good choice:

sudo apt install sendmail

Now, you should get an email anytime someone uses sudo!

Improve Server Stability

VPS servers can run out of memory during traffic spikes or other system events. In this situation, the server might go into “swap hell”. It’s important to configure your applications so memory swapping does not occur.

Modern servers like Nginx or Node.js use a single process to handle multiple simulateous connections, so this is less of a problem than in the past.

For example, in Apache 2.2.x (quite old), the default settings allowed 150 clients to connect simultaneously. This was way too large a number for a typical small VPS server. Let’s do the math. Apache’s processes were typically ~25MB each. If the website got a temporary traffic spike and 150 processes launched, we’d need 3750MB of memory on the server. If we don’t have this much, then the OS will grind to a halt as it swaps memory to disk to make room for new processes, but then immediately swaps the stuff on disk back into memory. This is also known as “swap hell”.

No useful work gets done once “swap hell” occurs. The server can be stuck in this state for hours, even after the traffic rush has subsided. During this time, very few web requests will get serviced.

If you’re still using the ancient Apache 2.2.x for some reason, you could set MaxClients to something more reasonable like 20 or 30 clients. There are many other optimizations to make, too. Linode has a Library article with optimizations for various server types.

Newer version of Apache (2.4 and up) use an “event based mpm” instead of Apache 2.2 ineffecient “prefork” approach. This is far less of a problem with the improved approach.

And of course, servers like Nginx and Node.js handle thousands of connections without making a new process for each connection.

Reboot server on out-of-memory condition

In cases where something goes awry, it is good to automatically reboot your server when it runs out of memory. This will cause a minute or two of downtime, but it’s better than languishing in the swapping state for potentially hours or days.

You can leverage a couple kernel settings and Lassie to make this happen on Linode.

Adding the following two lines to your /etc/sysctl.conf will cause it to reboot after running out of memory:

vm.panic_on_oom=1
kernel.panic=10

The vm.panic_on_oom=1 line enables panic on OOM; the kernel.panic=10 line tells the kernel to reboot ten seconds after panicking.

Read more about rebooting when out of memory on Linode’s wiki.

Miscellaneous nice-to-haves

These next things are not required but are nice to do.

Set up reverse DNS

The reverse DNS system allows one to determine the domain name that lives at a given IP address. This is useful for network troubleshooting — (ping, traceroute, etc.), as well as email anti-spam measures (read more on Wikipedia).

It’s pretty easy to set up. From the Linode Manager, select your Linode, click on “Remote Access”, then click on “Reverse DNS” (under “Public IPs”). Type in your domain and that’s it!

Set up a private IP address

Private IPs are useful for communicating data on the Linode network, i.e. Linode to Linode. This is handy if you have multiple Linodes (say, one for your web server and one for your database). Private network traffic is more secure (only other Linode customers can see it, vs. the whole internet), faster (the traffic never has to leave the datacenter if both Linodes are in the same datacenter), and free (doesn’t count towards your monthly bandwidth quota).

I currently put my database server on its own Linode, so that I can scale it independently of my frontend servers and debug performance issues easier since the systems are isolated. This hasn’t been super-handy yet, but if one of my sites gets a huge traffic rush, I bet it will be immensely useful.

It’s easy to set up. On the Remote Access tab, click Add a Private IP.

Then, just restart the Linode and the new IP address will become available thanks to Linode Network Helper which automatically deposits a static networking configuration in to your Linode at boot.

Configuring your applications and your database to route traffic over the local network is another issue, not covered here.

Install Useful Server Software

At this point, you have a pretty nice server setup. Congrats! But, your server still doesn’t do anything useful. Let’s install some software.

Install a compiler

A compiler is often required to install Python packages and other software, so let’s just install one up-front.

sudo apt install build-essential

Install Nginx

sudo apt install nginx

Install Node.js

Follow the instructions to install the NodeSource Node.js PPA.

Install MySQL

Install MySQL:

sudo apt install mysql-server libmysqlclient-dev

Set root password when prompt asks you.

Verify that MySQL is running.

sudo netstat -tap | grep mysql

For connecting to MySQL, instead of the usual PHPMyAdmin, I now use Sequel Pro, a free app for Mac.

Improve MySQL security

Before using MySQL in production, you’ll want to improve your MySQL installation security. Run:

mysql_secure_installation

This will help you set a password for the root account, remove anonymous-user accounts, and remove the test database.

Keep your MySQL tables in tip-top shape

Over time your MySQL tables will get fragmented and queries will take longer to complete. You can keep your tables in top shape by regularly running OPTIMIZE TABLE on all your tables. But, since you’ll never remember to do this regularly, we should set up a cron job to do this.

Open up your crontab file:

crontab -e

Then, add the following line:

@weekly mysqlcheck -o --user=root --password=<your password here> -A

Also, you can try manually running the above command to verify that it works correctly.

Backup your MySQL databases

The excellent automysqlbackup utility can automatically make daily, weekly, and monthly backups of your MySQL database.

Install it:

sudo apt install automysqlbackup

Now, let’s configure it. Open the configuration file:

sudo vim /etc/default/automysqlbackup

By default, your database backups get stored in /var/lib/automysqlbackup which isn’t very intuitive. I recommend changing it to a folder within your home directory. To do this, find the line that begins with BACKUPDIR= and change it to BACKUPDIR="/home/<your username>/backups/mysql"

You also want to get an email if an error occurs, so you’ll know if automatic backups stop working for some reason. Find the line that begins with MAILADDR= and change it to MAILADDR="<your email address>".

Close and save the file. That’s it!

Setup Automatic Backups

Backups are really important. Linode offers a paid backup service that’s really convenient if you accidentally destroy something and need to restore your Linode quickly. It’s $5 per month for the smallest Linode. I enable it on all my Linodes.

If you want even more peace of mind (or don’t want to pay for Linode’s backup service) you can roll your own simple backup solution using rsync.

You will need access to another Linux server (maybe another Linode?) or a home server. I just installed Ubuntu on an old desktop computer to use as a backup server.

We’re going to create a weekly cronjob that backs up our Linode’s home directory to a backup server. I keep all the files that I would want to backup in my home folder, so this works for me.

Open your crontab:

crontab -e

Add this line to the file:

@weekly rsync -r -a -e "ssh -l <your username on backup server> -p <ssh port number of backup server>" --delete /home/<your username> <hostname or ip address of backup server>:/path/to/some/directory/on/backup/server

I recommend running the above command manually to make sure you have it right before adding it to your crontab file.

That’s it!

Linode rocks!

If, after reading this, you want to sign up for Linode, use this link and I’ll get a couple weeks of free hosting. If you prefer not to, here’s the plain link: Linode.com

Happy hacking!

Using functional magnetic resonance imaging (fMRI), it is possible to see your decisions before you realize that you’ve made them. Researchers asked people to press a button using their left or right index fingers while monitoring their real time brain activity using fMRI. The results were surprising:

The conscious decision to push the button was made about a second before the actual act, but the team discovered that a pattern of brain activity seemed to predict that decision by as many as seven seconds. Long before the subjects were even aware of making a choice, it seems, their brains had already decided. (from Nature)

These results are shocking. They imply that all decisions are out of our control, subject to the whims of the gray matter in our heads.

I spent a lot of time thinking about this (always a dangerous pastime, hehe). I came up with a programming metaphor that makes a pretty convincing argument against the existence of free will. I thought it was interesting enough to be worth sharing, so here we go:

The human mind is a self-modifying function

Every human mind is a function. We take as input the sensory experience of the world around us. And we produce as output the electric signals that control our body and let us take action.

Of course, we don’t control the parameters that we are invoked with. The world outside, the people we meet, the problems we face are all inputs that we (as functions) must consume; they are things that we must make decisions about – how do I react when my friend says hi? how do I react when I see a car driving directly at me? etc…

However, humans are not merely normal functions. We can change through learning. Thus, we are self-modifying functions – we have the ability to re-write ourselves based on the input we receive.

Learning

If we touch a stove and burn our hand, we use our knowledge of causation (i.e. touching the hot stove caused pain), past experiences with pain (pain is something to avoid), and learning strategies (how to process new information into a useful format for storage) to modify ourselves. Thus, the next time we are presented with a hot stove we won’t decide to touch it a second time.

Interestingly, this is not true for everyone. Some people will still touch the stove a second time and burn themselves. Why? Because their memory of past experiences, their learning tools, or something else in their mind isn’t working correctly. Their function is buggy and for whatever reason doesn’t have the mechanism to rewrite itself properly (i.e. learn).

So, continuing with the analogy of humans as self-modifying functions, we have, in some sense, the ability to change ourselves. But, in the end, it’s still the contents of the function body which determines how we react to the input we are given. The function body decides how to process the input, and thus, what we learn from the experience.

Thus, the tools you currently have in your brain are inherently limiting. You can turn inputs into permanent knowledge, but only using the tools and knowledge you have available to you now. So, the pseudo-code for your brain would be an event loop that looks like this:

var brain = function() { /* Your genes define your initial brain state */ };

while (true) {
  brain = brain( get_sensory_input() ); // Returns a new function that is different from before
  brain.act(); // Act based on your current brain state
}

You have no free will

When you think about it like this, you see that your current brain state determines your next brain state. Thus, your current brain state was determined by your previous brain state. Using induction, you can take this all the way back to earliest childhood.

Even if you try to improve or change yourself through learning, meditation, or introspection, you are still limited by the physiology of your brain. For example, even if you introspect really intensely, you can’t really ever free yourself of your inherent limitations. Even thoughts about thought itself (i.e. metacognition) are still thoughts, controlled by the physiology of your brain, your past experiences, your emotional state, etc. Metacognition and introspection create a convincing illusion of free will.

Thus, I claim that the cognitive “function” you were born with, combined with inputs from your environment, ultimately determines who you are today. Since you do not determine your own starting state, nor the inputs you receive, you aren’t really responsible for your current self, your achievements, or your faults. In the end, all mental processes are out of your control.

What are the implications of this?

None, really. This discussion isn’t particularly practical since there is no easy method to verify the validity of any of these claims. It’s just interesting to think about.

I don’t think that believing or disbelieving in free will should change your behavior. In fact, believing that you don’t have any free will could probably only change your behavior for the worse. Discovering the true nature of free will is important not for any practical reason, but for the same reason that discovering how any other part of reality works is important. It brings us closer to the truth, to understanding our lives and our place in the universe.

More on this subject

If you want to think about this stuff more, I highly recommend this talk: Sam Harris on “Free Will”. It is brilliant. Sam lays out all the main arguments against the existence of free will in a clear, persuasive manner.

Also, I recently watched a Google Tech Talk that contained a lot of great information on how brains in young children work. It’s called Think faster, focus better, and remember more. It only strengthens my belief in this self-modifying function hypothesis.

Lastly, the New York Times recently published this: Neuroscience and Moral Responsibility.

If you have any thoughts on this post, please leave a comment. I’m interested to hear what others think about this idea.

Update (Aug 12, 2012)

My friend Mikee Nguyen sent me this excellent, relevant clip from the film Waking Life:

Today, I’m releasing the source code for a couple of my past projects.

The projects

• Instant.fm, a music playlist sharing service. [github]
• Fling, a way to send songs, videos, and urls from your phone to your desktop computer by flicking your wrist. [github]

I released the code for these projects a while ago but never posted about it:

• CMSploit, a security scanner to find temporary config files that contain passwords on public websites. [github]
• WireSheep, shows you each user on the network and all the HTTP requests they’re making. [github]
• Selective Attention Test, a new version of the famous gorilla video. [github]

Your turn!

Release some of your own code! All the cool kids are doing it. Happy hacking!

TL;DR – Summary of the Problem

Using a text editor to modify content management system (CMS) configuration files (like wp-config.php) could expose your database password to the world. Several popular text editors like Vim and Emacs automatically create backup copies of the files you edit, giving them names like wp-config.php~ and #wp-config.php#. If the text editor crashes or the SSH connection drops during editing, then the temporary backup files may not be cleaned up correctly. This means that the CMS config file (which contains the database password) could accidentally be made public to anyone who knows where to look.

Most servers, including the ubiquitous Apache, will happily serve the plaintext of .php~ and .php# files without passing them through the PHP preprocessor first, since they don’t have the .php file extension. Thus, your sensitive database credentials are just one GET request away from being accessed by a malicious party.

I wrote an automatic program, which I call CMSploit, to test for the prevalence of this issue across the wider web. I tested the top 200,000 websites (as ranked by Quantcast) and found that 0.11% of websites are vulnerable. If we eliminate non-CMS sites, and just look at CMS-powered websites, then we find that 0.77% of websites running a CMS have publicly-visible config files.

Here is a presentation I gave to the Stanford ACM about CMSploit, including screenshots of sites whose passwords I obtained using my CMSploit tool. Or, if you want all the gory details, then keep reading.

CMS Configuration Files

Most content management systems (CMSs) store sensitive settings like the database hostname, database name, database username, and database password in a file that sits in the root of the web directory.

Here is what a typical config file looks like:

<?php

define('DB_NAME', 'my_secret_database'); /** Name of WordPress database */
define('DB_USER', 'secret_agent_1'); /** MySQL database username */
define('DB_PASSWORD', 'you_will_never_guess_this'); /** MySQL database password */
define('DB_HOST', 'localhost'); /** MySQL hostname */

?>

Here is a list of the various configuration files used by the most popular CMSs:

wp-config.php        # WordPress
config.php           # phpBB, ExpressionEngine
configuration.php    # Joomla
LocalSettings.php    # MediaWiki
mt-config.cgi        # Movable Type
settings.php         # Drupal

Despite the fact that these configuration files exist in a publicly accessible folder, the file contents are unviewable by a normal web user. Accessing the file directly does not work because the PHP interpreter handles all requests to .php files and wisely returns a blank page instead of the actual file contents. (Try accessing http://www.example.com/wp-config.php on your favorite WordPress blog to see what I mean. You should get a blank page back.)

As you can see, all the sensitive database information is located within <?php ?> tags. So, even if a malicious user were to access your config file directly, the PHP preprocessor would just run the PHP code, which defines some PHP global variables and then returns a blank page. Thus, no harm done, right?

Text Editors Make Temporary Files

Popular command line text editors like Vim, Emacs, Gedit, and Nano create several temporary backup files during the course of file editing. When you open a file for editing, a backup of the original file is saved. Depending on your text editor, in-progress file changes might also be saved to a swap file, so you can restore your unsaved changes in the event of a program crash, power outage, or connectivity issue.

If all goes well, when you’re done editing the file, the text editor deletes the temporary files so your filesystem doesn’t accumulate dozens of old temporary files. However, if your text editor crashes or you lose your connection, then the temporary files will still be on your filesystem.

Here are the temporary filenames used by the most popular text editors (assuming a file named wp-config.php):

wp-config.php~        # Vim, Gedit
#wp-config.php#       # Emacs
wp-config.php.save    # Nano
wp-config.php.swp     # Vim (swap file)
wp-config.php.swo     # Vim (swap file)

Putting Two and Two Together

If a CMS user edits a config file on their live site (as opposed to editing it offline and uploading it over FTP), then there may be temporary files which contain their database password floating around in publicly-accessible folders.

If someone requests one of these temp files, then most servers will return the plaintext, skipping the PHP parser completely – yikes! By default, Apache assumes that only files which have a .php file extension are PHP files. If the file extension is not .php, Apache happily serves up the plaintext of the file.

How prevalent is this problem?

After noticing this security issue on one of my websites, I became curious to find out how common it is across the wider web. So, I wrote a program to test the top websites and get a rough idea of the prevalence of this problem. I call it CMSploit. The program is pretty simple – it issues GET requests to a site to test for the presence of temporary backup files with common CMS config filenames.

Here were my results:

• Tested the 216,391 most popular websites (according to Quantcast).
• Found 230 config files visible in root of site.
• Thus, 230 / 216391 = 0.11% of all websites are vulnerable.
• Latest stats say that about 13.8% of the top 10,000 websites run CMSs. If we just focus on CMS-powered websites, then the percentage of vulnerable sites is much higher:
• Thus, 230 / (216391 * 0.138) = 0.77% of websites running a CMS are vulnerable.

It’s shocking to think that 0.77% of websites (1 out of every 130) built with a CMS has its database password just sitting there in a public folder for all the world to see. Lots of these are popular, active websites. You would likely recognize many of them. Most of the sites were WordPress blogs, but there were a surprising number of e-commerce sites too, which is a little scary.

Responsible Disclosure

1. I contacted several of the highest profile sites to notify them of this security issue on their site. Most of them fixed the problem within a few days. All who replied to me were extremely grateful for bringing the issue to their attention. One of the companies even offered me a free license to their software.

2. I submitted a vulnerability report with US-CERT. Unfortunately, they replied with “This issue is not the type of vulnerability class we are inclined to coordinate or publish on.” I also plan to submit reports with Apache, PHP, WordPress, and Vim/Emacs.

3. After running the script and collecting my research statistics (published above), I securely wiped all the config files from my hard disk. I did not attempt to login with any of the database credentials I discovered. Therefore, it was not possible to determine what percentage of the database credentials were valid or what percentage of database servers were open to remote connections.

Lessons Learned

1. CMS users should never edit their config.php file (or other sensitive files) with a text editor that creates temporary backup files. The best policy is to avoid editing any sensitive files on a live website. Instead, copy the file locally, make your edits to it, and copy it back to the server.

2. It’s trivially easy to write a script to search for vulnerable sites. Bad people have probably been doing it for several years. In fact, this issue has even been discussed in other forums before. You should check your sites for wp-config.php~ and related files. Make sure your sites are not vulnerable.

3. Someone should fix this. It’s not completely clear where responsibility lies, though. Is it Apache’s fault? Or PHP? Or vim/emacs? Should WordPress and other CMSs do something about it? There are many ways to fix this problem. I don’t particularly care how it gets fixed, as long as the default configuration of Apache + PHP + vim/emacs + WordPress don’t have this problem, I’ll be satisfied. In the meantime, using this very common web stack we have a scenario where ~1% of sites expose their passwords. This is bad.

4. You should configure MySQL to deny remote connections to your database and connect to localhost instead. If you absolutely need remote access, then explicitly whitelist certain IPs and deny the rest. This way, if someone gets your database credentials, they’ll be unable to actually log in.

5. In the short term, you should proactively protect all your websites. If you run WordPress and Apache, you can block access to any file containing the string wp-config.php with a .htaccess rule like this:

<Files ~ "(^#.*#|~|\.sw[op])$">
Order allow,deny
Deny from all
</Files>

Modify the rule to suit your CMS and server environment.

Final Thoughts

Even though the discovery that 0.77% of CMS-powered websites have public database passwords is already shocking, I’m pretty confident that you could easily double or triple the number of vulnerable sites with a better, more thorough script, and lots more time.

The script I wrote only tests the root of each site for CMS config files. However, lots of sites run CMSs in subfolders and subdomains like /blog/, /wiki/, /forums/, blog.mydomain.com, etc. Testing these places would dramatically increase the number of vulnerable sites detected.

I will not publish the source code of this script, because of the potential for harm. However, if you are a security researcher and are interested in reviewing the source code, send me an email.

This DEFCON talk is relevant: Pillaging distributed version control system repos for fun and profit.

Discussion on reddit.

Update (Mar 8 2012)

There is now an Nmap script, inspired by CMSploit, that appears to do everything that my script does. (I’m not affiliated with Nmap or this particular script.)

Update (Aug 22 2012)

The code for CMSploit is now available on Github.

Thanks to John Hiesey for reading a draft of this.

The answer to both questions is Linode, so here is my review of Linode hosting.

Linode = Awesome Hosting

I’ve been using Linode to host my websites for the past 5+ years and I’m thoroughly impressed by the service. I can’t recommend them enough – it is easily the best web hosting service I’ve used to date.

If you’re coming from the shared hosting world, then let me tell you: a virtual private server (VPS) is a whole new world. A VPS means freedom, power, and total flexibility. You get access to everything from root, the kernel, and on up. All managed from the command line and, occasionally, the simple Linode control panel. Your Linode is completely and totally your machine.

Linode uses the Xen hypervisor for virtualization which means that your CPU, memory, and disk resources are really yours. You get a guaranteed portion of the server resources. If another user has a really popular site, your sites will not be affected. However, if there are extra unused resources, then your websites will automatically burst above your limits to make use of the extra CPU cycles or disk time, if you require them.

A couple things to keep in mind about Linode:

1. Your Linode comes with a vanilla installation of Linux. You get to pick which Linux flavor to install on your server (I chose Ubuntu), but you have to configure the whole thing from the command line. Which brings us to…

2. Your Linode comes with no traditional control panel to help you set up your domains or help you configure virtual hosting. That means no CPanel or Plesk. Just the command line.

   You will need to install Apache, MySQL, set up iptables firewall rules, disable remote root access, and configure Apache virtual hosts (to serve multiple sites with one IP address) etc. on your own. At first, this scared me a bit, but it’s actually not bad at all. It took me about a day to learn how to do everything I wanted to do and there are plenty of excellent tutorials online. The Linode Library and Slicehost Articles should have all you need to get going.

Some things that surprised me:

1. I get stuff done much more quickly on the command line then I ever did using control panels and lame, poorly designed graphical tools.

2. The command line is extremely empowering. If you’ve never had to use the command line for much before, it can be intimidating. At first, it’s difficult to understand the appeal of the command line, especially if you’re a designer or a programming noob. But, trust me, the command line is the way to go. The sooner you get comfortable using it, the happier you’ll be. I’m telling you, it’s a whole new world.

Another advantage of using the command line is that you no longer need to buy hosting that offers a bloated control panel package like CPanel or Plesk. Because of this, Linode prices are dirt cheap: $20/month to get started.

Some things that make Linode awesome:

1. About once a year, Linode upgrades the storage and RAM for all users at no extra cost. For example: when I first joined Linode in 2009, I hosted all my sites on their cheapest plan – 360MB ram, 16GB storage, 200GB transfer – for $20/month. Since then, they’ve upgraded this so the same server now comes with 1GB ram, 48GB storage, and 2TB transfer – neat!

2. Unlike most shared hosting providers, Linode does not oversell their hosting. At a cheap shared host, if you get even close to using all the bandwidth and storage that you paid for, you’d get shut down for “misusing resources” and “unfairly affecting other users on the server”. With Linode, your resources are actually yours – Linode doesn’t oversell.

Hosting Viral Websites with Linode

If you’re considering a Linode, I recommend you start out with the smallest one and upgrade if/when you need to.

I currently host all my sites on the “Linode 1GB” plan, which is probably overkill, but its nice to be able to handle random traffic spikes without much work. When a site goes viral, like YouTube Instant did or EmuSpin did, I just migrate to a bigger Linode.

For example, as YouTube Instant picked up steam, I upgraded the server several times, eventually ending up at the “Linode 4GB” plan – at the time, 4GB ram, 160GB storage, and 1600GB transfer. Migrating plans takes about 15 minutes, and your sites will be inaccessible during that time.

If I had more time, I would have switched from Apache to Nginx so that each new visitor wouldn’t spawn a whole new 20MB process. As you’ll quickly find out, the scarcest resource in a VPS hosting environment is RAM. Nginx, node.js – and other high concurrency, low memory servers – are a lifesaver here.

Linode rocks!

If you want to sign up, use my referral link and I’ll get a couple weeks of free hosting. If you prefer not to, here’s the plain link: Linode.com

Happy hacking!

Delving back into project history long before we launched publicly in 2008, the autoupdate project was one of the very first we started working on. The idea was to give people a blank window with an autoupdater. If they installed that, over time the blank window would grow into a browser. And today, some five years after our autoupdater started updating a mostly blank window that could barely load webpages, it is now an engine for delivering an incredibly sophisticated web technology platform onto our users’ computers, which in turn allows web app developers to build amazing new online experiences. I have never seen such an effective platform update mechanism before.

How cool is that??! :)

Invisibility is a key part of one of Chrome’s four “S”s - simplicity. We seek not to bother the user with information that is irrelevant. As important as it might seem to engineers for our users to weigh fully a variety of considerations relating to our software, in reality they have better things to be doing. They’re trying to find a restaurant, check an email, post a facebook update, and then get the hell away from the computer. Every time the user agent steps in between the user and their task, we slow them down and diminish their enjoyment. Content, not Chrome.

What an awesome way to think about design.

It works in all versions of Adobe Flash that I tested. I’ve confirmed that it works in the Firefox and Safari for Mac browsers. Use one of those if you check out the live demo. There’s a weird CSS opacity bug in most other browsers (Chrome for Mac and most browsers on Windows/Linux).

Video demo of the attack

Source code: Github. Video demo: YouTube

Updates about the vulnerabilty

• 10/19/2011: CNET says that Adobe is working on a fix and it could be ready by end of week. Adobe also emailed me and said “our product team is wrapping up their investigation and is now working on a fix, which should not require a Flash Player update”.

• 10/20/2011: Whoa, this story is everywhere! News stories have been published in CNET, Wired.com, The Register, Ars Technica, Gizmodo, PC World, Yahoo! News, ZDNet (and another ZDNet), The Inquirer, Computer World, and The H Security. Lastly, this is an interesting opinion piece: “The Sins of the Flash”

• 10/20/2011: Adobe says they just posted a fix to the Settings Manager that should resolve the issue. I just tested it out, and indeed the issue appears to be fixed now. Congrats, Adobe, for the quick fix!

• 12/21/2011: This attack made it into Jeremiah Grossman’s list of top web hacking techniques of 2011. It’s #26.

• 1/10/2012: Another similar clickjacking attack was just discovered. Adobe has fixed it.

• 5/9/2012: FOX News in Cleveland just ran a story about this.

Read on for the original blog post.

Clickjacking + Adobe Flash = Sad Times!

This attack works by using a neat variation of the normal clickjacking technique that spammers and other bad people are using in the wild right now. For the uninitiated:

Clickjacking is a malicious technique of tricking Web users into revealing confidential information or taking control of their computer while clicking on seemingly innocuous web pages.

– Wikipedia

Combine clickjacking with the Adobe Flash Player Setting Manager page and you have a recipe for some sad times.

Background

I took a computer security class (Stanford’s CS 155) last quarter and really enjoyed this research paper on framebusting and clickjacking. After reading it, I checked out a few popular sites to see if it was possible to clickjack them. After a couple hours, I had no success.

But, then I stumbled upon this blog post entitled “Malicious camera spying using ClickJacking” where the author shows how to clickjack the Adobe Flash Settings Manager page to enable users’ webcams. He accomplishes this by putting the whole settings page into an iframe and making it invisible. Then, unsuspecting users play a little game and unwittingly enable their webcams. Adobe quickly added framebusting code to the Settings Manager page (why wasn’t it there in the first place?), and the attack stopped working.

But alas, the same attack is actually still possible.

How my attack works

Instead of iframing the whole settings page (which contains the framebusting code), I just iframe the settings SWF file. This let me bypass the framebusting JavaScript code, since we don’t load the whole page – just the remote .SWF file. I was really surprised to find out that this actually works!

I’ve seen a bunch of clickjacking attacks in the wild, but I’ve never seen any attacks where the attacker iframes a SWF file from a remote domain to clickjack it – let alone a .SWF file as important as one that controls access to your webcam and mic!

The problem here is the Flash Player Setting Manager, this inheritance from Macromedia might be the Flash Player security Achilles heel.

– Guy Aharonovsky

This is a screenshot of what the Settings Manager .SWF file looks like:

Live Demo

I built a quick proof-of-concept demo to show how it works.

Important point: The demo is only guaranteed to work in Firefox and Safari for Mac. Right now, it doesn’t work in most other browsers since you can’t change the opacity or the z-index of an iframed swf file. However, I discovered a workaround that involves multiple iframes, but haven’t implemented it yet since it’s a bit complicated. But, I’m pretty sure that it’s possible to make it work everywhere, given enough time.

Important point 2: The vulnerability has been fixed by Adobe, so the demo does not work anymore.

View the demo. The code is also available on Github.

I should also mention that my demo builds heavily off of the ideas and work done by the dude who runs this blog, Guy Aharonovsky.

Also: If you’re a bit leery about running the demo… I promise I’m not saving the webcam video. I just display it back to you so you can see that it works. However, if an attacker used this technique, they would almost certainly NOT show you any sign that your cam is on. You’re only hope of finding out that something’s up is your webcam indicator light (if you have one).

Why release this?

I reported this vulnerability to Adobe a few weeks ago through the Stanford Security Lab. It’s been a few weeks and I haven’t heard anything from Adobe yet. I think it’s worth sharing it with the world now, so that Adobe pays attention and fixes it more quickly.

Although every browser and OS is theoretically susceptible to this attack, the process to activate the webcam requires multiple highly targeted clicks, which is difficult for an attacker to pull off. I’m not sure how useful this technique would actually be in the wild, but I hope that Adobe fixes it soon so we don’t have to find out.

Further reading

If you want to learn more about clickjacking and framebusting, you should read the excellent Busting Frame Busting: a Study of Clickjacking Vulnerabilities on Popular Sites (PDF) paper by Gustav Rydstedt, Elie Bursztein, Dan Boneh, and Collin Jackson.

Almost everything–all external expectations, all pride, all fear of embarrassment or failure–these things just fall away in the face of death, leaving only what is truly important. Remembering that you are going to die is the best way I know to avoid the trap of thinking you have something to lose. You are already naked. There is no reason not to follow your heart.

– Steve Jobs

One of my favorite ads:

Really gets the blood pumping, huh?

Interesting story about this video. Steve Jobs narrated the very first version of this Think Different commercial “Here’s to the Crazy Ones” (shown above). Sadly, this version never aired. Richard Dreyfuss did the voiceover in the version that aired.