R – sed to replace random site urls in ifrrame

First off, rolling your own regexp to parse URLs is a terrible idea. You must imagine this is a common enough problem that someone has written, debugged and tested a library for it, according to the RFCs. URIs are complex - check out the code for URL parsing in Node.js and the Wikipedia page on URI schemes.

There are a ton of edge cases when it comes to parsing URLs: international domain names, actual (.museum) vs. nonexistent (.etc) TLDs, weird punctuation including parentheses, punctuation at the end of the URL, IPV6 hostnames etc.

I've looked at a ton of libraries, and there are a few worth using despite some downsides:

• Soapbox's linkify has seen some serious effort put into it, and a major refactor in June 2015 removed the jQuery dependency. It still has issues with IDNs.
• AnchorMe is a newcomer that claims to be faster and leaner. Some IDN issues as well.
• Autolinker.js lists features very specifically (e.g. "Will properly handle HTML input. The utility will not change the href attribute inside anchor () tags"). I'll thrown some tests at it when a demo becomes available.

Libraries that I've disqualified quickly for this task:

• Django's urlize didn't handle certain TLDs properly (here is the official list of valid TLDs. No demo.
• autolink-js wouldn't detect "www.google.com" without http://, so it's not quite suitable for autolinking "casual URLs" (without a scheme/protocol) found in plain text.
• Ben Alman's linkify hasn't been maintained since 2009.

If you insist on a regular expression, the most comprehensive is the URL regexp from Component, though it will falsely detect some non-existent two-letter TLDs by looking at it.

Try tail:

tail -n +2 "$FILE"

-n x: Just print the last x lines. tail -n 5 would give you the last 5 lines of the input. The + sign kind of inverts the argument and make tail print anything but the first x-1 lines. tail -n +1 would print the whole file, tail -n +2 everything but the first line, etc.

GNU tail is much faster than sed. tail is also available on BSD and the -n +2 flag is consistent across both tools. Check the FreeBSD or OS X man pages for more.

The BSD version can be much slower than sed, though. I wonder how they managed that; tail should just read a file line by line while sed does pretty complex operations involving interpreting a script, applying regular expressions and the like.

Note: You may be tempted to use

# THIS WILL GIVE YOU AN EMPTY FILE!
tail -n +2 "$FILE" > "$FILE"

but this will give you an empty file. The reason is that the redirection (>) happens before tail is invoked by the shell:

1. Shell truncates file $FILE
2. Shell creates a new process for tail
3. Shell redirects stdout of the tail process to $FILE
4. tail reads from the now empty $FILE

If you want to remove the first line inside the file, you should use:

tail -n +2 "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"

The && will make sure that the file doesn't get overwritten when there is a problem.