This is Section 2 of the sed FAQ. |
"sed" stands for Stream EDitor. Sed is a non-interactive editor, written by the late Lee E. McMahon in 1973 or 1974. A brief history of sed's origins may be found in an early history of the Unix tools, at <http://www.columbia.edu/~rh120/ch106.x09>.
Instead of altering a file interactively by moving the cursor on the screen (as with a word processor), the user sends a script of editing instructions to sed, plus the name of the file to edit (or the text to be edited may come as output from a pipe). In this sense, sed works like a filter -- deleting, inserting and changing characters, words, and lines of text. Its range of activity goes from small, simple changes to very complex ones.
Sed reads its input from stdin (Unix shorthand for "standard input," i.e., the console) or from files (or both), and sends the results to stdout ("standard output," normally the console or screen). Most people use sed first for its substitution features. Sed is often used as a find-and-replace tool.
sed 's/Glenn/Harold/g' oldfile >newfile
will replace every occurrence of "Glenn" with the word "Harold", wherever it occurs in the file. The "find" portion is a regular expression ("RE"), which can be a simple word or may contain special characters to allow greater flexibility (for example, to prevent "Glenn" from also matching "Glennon").
My very first use of sed was to add 8 spaces to the left side of a file, so when I printed it, the printing wouldn't begin at the absolute left edge of a piece of paper.
sed 's/^/ /' myfile >newfile # my first sed script sed 's/^/ /' myfile | lp # my next sed script
Then I learned that sed could display only one paragraph of a file, beginning at the phrase "and where it came" and ending at the phrase "for all people". My script looked like this:
sed -n '/and where it came/,/for all people/p' myfile
Sed's normal behavior is to print (i.e., display or show on screen) the entire file, including the parts that haven't been altered, unless you use the -n switch. The "-n" stands for "no output". This switch is almost always used in conjunction with a 'p' command somewhere, which says to print only the sections of the file that have been specified. The -n switch with the 'p' command allow for parts of a file to be printed (i.e., sent to the console).
Next, I found that sed could show me only (say) lines 12-18 of a file and not show me the rest. This was very handy when I needed to review only part of a long file and I didn't want to alter it.
# the 'p' stands for printsed -n 12,18p myfile
Likewise, sed could show me everything else BUT those particular lines, without physically changing the file on the disk:
# the 'd' stands for deletesed 12,18d myfile
Sed could also double-space my single-spaced file when it came time to print it:
sed G myfile >newfile
If you have many editing commands (for deleting, adding, substituting, etc.) which might take up several lines, those commands can be put into a separate file and all of the commands in the file applied to file being edited:
# 'script.sed' is the file of commands # 'myfile' is the file being changedsed -f script.sed myfile # 'script.sed' is the file of commands
It is not our intention to convert this FAQ file into a full-blown sed tutorial (for good tutorials, see section 2.3). Rather, we hope this gives the complete novice a few ideas of how sed can be used.
Note: "Free" does not mean "public domain" nor does it necessarily mean you will never be charged for it. All versions of sed in this section except the CP/M versions are based on the GNU general public license and are "free software" by that standard (for details, see http://www.gnu.org/philosophy/free-sw.html). This means you can get the source code and develop it further.
At the URLs listed in this category, sed binaries or source code can be downloaded and used without fees or license payments.
ssed is the version recommended by the FAQ maintainers, since it shares the same codebase with GNU sed, has the most options, and is free software (you can get the source). Though there were earlier version of ssed distributed, sites for these are not being listed.
http://sed.sourceforge.net/grabbag/ssed http://freshmeat.net/project/sed/
This is the latest official version of GNU sed. It offers in-place text replacement as an option switch.
ftp://ftp.gnu.org/pub/gnu/sed/sed-4.0.5.tar.gz http://freshmeat.net/project/sed
Based on the latest version of GNU sed, which supports multi-byte characters.
ftp://ftp1.freebsd.org/pub/FreeBSD/FreeBSD-stable/packages/Latest/ja-sed.tgz
An alpha test release which was the base for the development of ssed and GNU sed v4.0.
ftp://alpha.gnu.org/pub/gnu/sed/sed-3.02.80.tar.gz
Interim version with most features of GNU sed v3.02.80.
ftp://ftp.gnu.org/pub/gnu/sed/sed-3.02.tar.gz
source code and binaries for Debian GNU/Linux
http://www.debian.org/Packages/stable/base/sed.html
For some time, the GNU project <http://www.gnu.org> used Eric S. Raymond's version of sed (ESR sed v1.1), but eventually dropped it because it had too many built-in limits. In 1991 Howard Helman modified the GNU/ESR sed and produced a flexible version of sed v1.5 available at several sites (Helman's version permitted things like \<...\> to delimit word boundaries, \xHH to enter hex code and \n to indicate newlines in the replace string). This version did not catch on with the GNU project and their version of sed has moved in a similar but different direction.
http://catb.org/~esr/sed-1.3.tar.gz
Eric Raymond <esr@snark.thyrsus.com> wrote one of the earliest versions of sed. On his website <http://www.catb.org/~esr/> which also distributes many freeware utilities he has written or worked on, he describes sed v1.1 this way:
"This is the fast, small sed originally distributed in the GNU toolkit and still distributed with Minix. The GNU people ditched it when they built their own sed around an enhanced regex package -- but it's still better for some uses (in particular, faster and less memory-intensive)." (Version 1.3 fixes an unidentified bug and adds the L command to hexdump the current pattern space.)
http://www2s.biglobe.ne.jp/~vtgf3mpr/gnu/sed.htm
http://hobbes.nmsu.edu/pub/os2/util/file/sed-3_02-r2-bin.zip # binaries http://hobbes.nmsu.edu/pub/os2/util/file/sed-3_02-r2.zip # source
32-bit binaries and docs. Precompiled versions not available (yet).
32-bit binaries and docs, using DJGPP compiler. For details on new features, see Unix section, above.
http://www.student.northpark.edu/pemente/sed/sed3028a.zip # DOS binaries ftp://alpha.gnu.org/pub/gnu/sed/sed-3.02.80.tar.gz # source ftp://ftp.simtel.net/pub/simtelnet/gnu/djgpp/v2gnu/sed3028b.zip # binaries ftp://ftp.simtel.net/pub/simtelnet/gnu/djgpp/v2gnu/sed3028d.zip # docs ftp://ftp.simtel.net/pub/simtelnet/gnu/djgpp/v2gnu/sed3028s.zip # source
32-bit binaries, no docs. Requires 80386 DX (SX will not run) and must be run in a DOS window or in a full screen DOS session under Microsoft Windows. Will not run in MS-DOS mode (outside Win/Win95). We recommend using the latest version of GNU sed.
http://www.simtel.net/pub/win95/prog/gsed205b.zip ftp://ftp.cdrom.com/pub/simtelnet/win95/prog/gsed205b.zip
modified by Frank Whaley.
This version was part of the "Virtually UN*X" toolset, hosted by itribe.net; that website is now closed. Gsed v1.03 supported Win9x long filenames, as well as hex, decimal, binary, and octal character representations.
http://www.cygwin.com
Formerly know as "GNU-Win32 tools." According to their home page, "The Cygwin tools are Win32 ports of the popular GNU development tools for Windows NT, 95 and 98. They function through the use of the Cygwin library which provides a UNIX-like API on top of the Win32 API." The version of sed used is GNU sed v3.02.
http://www.mingw.org http://mingw.sourceforge.net
According to their home page, "MinGW ('Minimalist GNU for Windows') refers to a set of runtime headers, used in building a compiler system based on the GNU GCC and binutils projects. It compiles and links code to be run on Win32 platforms ... MinGW uses Microsoft runtime libraries, distributed with the Windows operating system." The version of sed used is GNU sed v3.02.
Compiled with Mingw32 for 32-bit environments described above. This version should support Win95 long filenames.
http://www.dbnet.ece.ntua.gr/~george/sed/OLD/sed15.exe http://www.student.northpark.edu/pemente/sed/sed15exe.zip
This is a forthcoming version, now in beta testing, but with many new features. It corrects all the bugs in sed v1.5, and adds the best features of sedmod v1.0 (below). It is available in 16-bit and 32-bit compiled versions for MS-DOS. Sorry, no URLs available yet.
uncompiled source code (Turbo C)
ftp://ftp.simtel.net/pub/simtelnet/msdos/txtutl/sed15.zip ftp://ftp.cdrom.com/pub/simtelnet/msdos/txtutl/sed15.zip
DOS executable and documentation
ftp://ftp.simtel.net/pub/simtelnet/msdos/txtutl/sed15x.zip ftp://ftp.cdrom.com/pub/simtelnet/msdos/txtutl/sed15x.zip
http://www.ptug.org/sed/SEDMOD10.ZIP http://www.student.northpark.edu/pemente/sed/sedmod10.zip ftp://garbo.uwasa.fi/pc/unix/sedmod10.zip
See section 2.2.1.3 ("Microsoft Windows"), above.
Does not run under MS-DOS.
32-bit binaries and source, using DJGPP compiler. Requires 80386 SX or better. Also requires 3 CWS*.EXE extenders on the path. See section 5.5 ("What is CSDPMI*B.ZIP and why do I need it?"), below. We recommend using a newer version of GNU sed.
http://www.simtel.net/pub/simtelnet/gnu/djgpp/v2gnu/sed118b.zip ftp://ftp.cdrom.com/pub/simtelnet/gnu/djgpp/v2gnu/sed118b.zip http://www.simtel.net/pub/simtelnet/gnu/djgpp/v2gnu/sed118s.zip ftp://ftp.cdrom.com/pub/simtelnet/gnu/djgpp/v2gnu/sed118s.zip
16-bit binaries and source. Should run under any MS-DOS system.
http://www.simtel.net/pub/gnu/gnuish/sed106.zip ftp://ftp.cdrom.com/pub/simtelnet/gnu/gnuish/sed106.zip
Written for CP/M, ssed (for "small/stupid stream editor) supports only the a(ppend), c(hange), d(elete) and i(nsert) options, and apparently doesn't support regular expressions. A -u switch will "unsqueeze" compressed files and was used mainly in conjunction with DIF.COM for source code maintenance. (file: ssed22.lbr)
Rubenstein released a version of sed called CHANGE.COM (the TTOOLS.LBR archive member CHANGE.CZM is a "crunched" file). CHANGE.COM supports full RE's except grouping and backreferences, and its only function is global substitution. (file: ttools.lbr)
Since sed is a command-line utility, it is not customary to think of sed being used on a Mac. Nonetheless, the following instructions from Aurelio Jargas describe the process for running sed on MacOS version version 8 or 9.
(1) Download and install the Apple DiskCopy application
ftp://ftp.apple.com/developer/Development_Kits
(2) Download and install Apple MPW
ftp://ftp.apple.com/developer/Tool_Chest/Core_Mac_OS_Tools/MPW_etc./
(3) Download and expand Matthias Neeracher's GNU sed for MPW. (They seem to have misnumbered the sed filename.)
ftp://sunsite.cnlab-switch.ch/software/platform/macos/src/mpw_c/sed-2.03.sit.bin
(4) Enter the sed-3.02 directory and doubleclick the 'sed' file
(5) MPW Shell will open up. It will be a command window instead of a command line, but sed should work as expected. For example:
echo aa | sed 's/a/Z/g'<ENTER>
Note that ENTER is different from RETURN on an iMac. Apple also has its own version of sed on MPW, called "StreamEdit", with a syntax fairly similar to that of normal sed.
[ Additional information needed. ]
http://www.hamiltonlabs.com/cshell.htm
A sizable set of Unix/C shell utilities designed for OS/2. Price is $350 in the US, $395 elsewhere, with FedEx shipping, unconditional guarantee, unlimited support and free updates. A demo version of the suite can be downloaded from this site, but a stand-alone copy of sed is not available.
http://www.hamiltonlabs.com/cshell.htm
A sizable set of Unix/C shell utilities designed for Win9x, WinNT, and Win2K. Price is $350 in the US, $395 elsewhere, with FedEx shipping, unconditional guarantee, unlimited support and free updates. A demo version of the suite can be downloaded from this site, but a stand-alone copy of sed is not available.
http://www.interix.com
Interix (formerly known as OpenNT) is advertised as "a complete UNIX system environment running natively on Microsoft Windows NT", and is licensed and supported by Softway Systems. It offers over 200 Unix utilities, and supports Unix shells, sockets, networking, and more. A single-user edition runs about $200. A free demo or evaluation copy will run for 31 days and then quit; to continue using it, you must purchase the commercial version.
http://www.datafocus.com/products/nutc/
A different, yet related product line offered by MKS (Mortice Kern Systems, below); the awkward spelling "NuTCRACKER" is intentional. Various packages offer hundreds of Unix utilities for Win32 environments. Sed is not available as a separate product.
http://www.unixdos.com
UnixDos is a suite of 82 Unix utilities ported over to the Windows environments. There are 16-bit versions for Win3.x and 32-bit versions for WinNT/Win95. It is distributed as uncrippled shareware for the first 30 days. After the test period, the utilities will not run and you must pay the registration fee of $50.
Their version of sed supports "\n" in the RHS of expressions, and increases the length of input lines to 10,000 characters. By special arrangement with the owners, persons who want a licensed version of sed only (without the other utilities) may pay a license fee of $10.
http://www.research.att.com/sw/tools/uwin/
U/WIN is a suite of Unix utilities created for WinNT and Win95 systems. It is owned by AT&T, created by David Korn (author of the Unix korn shell), and is freely distributed only to educational institutions, AT&T employees, or certain researchers; all others must pay a fee after a 90-day evaluation period expires. U/WIN operates best with the NTFS (WinNT file system) but will run in degraded mode with the FAT file system and in further degraded mode under Win95. A minimal installation takes about 25 to 30 megs of disk space. Sed is not available as a separate file for download, but comes with the suite.
http://www.mixsoftware.com/product/utility.htm
According to their web page, "The C/Utilities Toolchest adds over 40 powerful UNIX utilities to your MS-DOS operating system. The result is an environment very similar to UNIX operating systems, yet 100% compatible with MS-DOS programs and commands." The toolchest costs $19.95, with source code available for an additional fee. Mix C's version of sed is not available separately.
http://www.mks.com
Sed comes bundled with the MKS Toolkit, which is distributed only as commercial software; it is not available separately.
http://www.tasoft.com
The Thompson Toolkit contains over 100 familiar Unix utilities, including a version of the Unix Korn shell. It runs under MS-DOS, OS/2, Win3.x, Win9x, and WinNT. Sed is one of the utilities, though Thompson is better known for its version of awk for DOS, TAWK. The toolkit runs about $150; sed is not available separately.
Sed & Awk, 2d edition, by Dale Dougherty & Arnold Robbins (Sebastopol, Calif: O'Reilly and Associates, 1997) ISBN 1-56592-225-5 http://www.oreilly.com/catalog/sed2/noframes.html
About 40 percent of this book is devoted to sed, and maybe 50 percent is devoted to awk. The other 10 percent covers regexes and concepts common to both tools. If you prefer hard copy, this is definitely the best single place to learn to use sed, including its advanced features.
The first edition is also very useful. Several typos crept into the first printing of the first edition (though if you follow the tutorials closely, you'll recognize them right away). A list of errors from the first printing of sed & awk is available at <http://www.cs.colostate.edu/~dzubera/sedawk.txt>, and errors in the 2nd are at <http://www.cs.colostate.edu/~dzubera/sedawk2.txt>, though most of these were corrected in later printings. The second edition tells how POSIX standards have affected these tools and covers the popular GNU versions of sed and awk. Price is about (US) $30.00
-----
Mastering Regular Expressions, 2d ed., by Jeffrey E. F. Friedl (Sebastopol, Calif: O'Reilly and Associates, 2002) ISBN 0-596-00289-0 http://regex.info http://www.oreilly.com/catalog/regex2/ http://public.yahoo.com/~jfriedl/regex/ (for the first edition)
Knowing how to use "regular expressions" is essential to effective use of most Unix tools. This book focuses on how regular expressions can be best implemented in utilities such as perl, vi, emacs, and awk, but also touches on sed as well. Friedl's home page (above) gives links to other sites which help students learn to master regular expressions. His site also gives a Perl script for determining a syntactically valid e-mail address, using regexes:
http://public.yahoo.com/~jfriedl/regex/code.html
-----
Awk und Sed, by Helmut Herold. (Bonn: Addison-Wesley, 1994; 288 pages) 2nd edition to be released in March 2003 ISBN 3-8273-2094-1 http://www.addison-wesley.de/main/main.asp?page=home/bookdetails&ProductID=37214
If you are interested in learning more about sed (its syntax, using regular expressions, etc.) you are welcome to subscribe to a sed-oriented mailing list. In fact, there are two mailing lists about sed: one in English named "sed-users", moderated by Sven Guckes; and one in Portuguese named "sed-BR" (for sed-Brazil), moderated by Aurelio Marinho Jargas. The average volume of mail for "sed-users" is about 35 messages a week; the average volume of mail for "sed-BR" is about 15 messages a week.
sed-BR mailing list: http://br.groups.yahoo.com/group/sed-br/ sed-users mailing list: http://groups.yahoo.com/group/sed-users/
To subscribe to sed-users, send a blank message to:
sed-users-subscribe@yahoogroups.com
To unsubscribe from sed-users, send a blank message to:
sed-users-unsubscribe@yahoogroups.com
The original users manual for sed, by Lee E. McMahon, from the 7th edition UNIX Manual (1978), with the classic "Kubla Khan" example and tutorial, in formatted text format:
http://sed.sourceforge.net/grabbag/tutorials/sed_mcmahon.txt
The source code to the preceding manual. Use "troff -ms sed" to print this file properly:
http://plan9.bell-labs.com/7thEdMan/vol2/sed http://cm.bell-labs.com/7thEdMan/vol2/sed
"Do It With Sed", by Carlos Duarte
http://www.dbnet.ece.ntua.gr/~george/sed/OLD/sedtut_1.html
"Sed: How to use sed, a special editor for modifying files automatically", by Bruce Barnett and General Electric Company
http://www.grymoire.com/Unix/Sed.html
U-SEDIT2.ZIP, by Mike Arst (16 June 1990)
ftp://ftp.cs.umu.se/pub/pc/u-sedit2.zip ftp://ftp.uni-stuttgart.de/pub/systems/msdos/util/unixlike/u-sedit2.zip ftp://sunsite.icm.edu.pl/vol/wojsyl/garbo/pc/editor/u-sedit2.zip ftp://ftp.sogang.ac.kr/pub/msdos/garbo_pc/editor/u-sedit2.zip
U-SEDIT3.ZIP, by Mike Arst (24 Jan. 1992)
http://www.student.northpark.edu/pemente/sed/u-sedit3.zip CompuServe DTPFORUM, "PC DTP Utilities" library, file SEDDOC.ZIP
Another sed FAQ
http://www.dreamwvr.com/sed-info/sed-faq.html
sed-tutorial, by Felix von Leitner
http://www.math.fu-berlin.de/~leitner/sed/tutorial.html
"Manipulating text with sed," chapter 14 of the SCO OpenServer "Operating System Users Guide"
http://ou800doc.caldera.com/SHL_automate/CTOC-Manipulating_text_with_sed.html
"Combining the Bourne-shell, sed and awk in the UNIX environment for language analysis," by Lothar Schmitt and Kiel Christianson. This basic tutorial on the Bourne shell, sed and awk downloads as a 71-page PostScript file (compressed to 290K with gzip). You may need to navigate down from the root to get the file.
ftp://ftp.u-aizu.ac.jp/u-aizu/doc/Tech-Report/1997/97-2-007.tar.gz available upon request from Lothar Schmitt <lothar@u-aizu.ac.jp>
http://sed.sourceforge.net/grabbag # Collected scripts http://main.rtfiber.com.tw/~changyj/sed/ # Yao-Jen Chang http://www.math.fu-berlin.de/~guckes/sed/ # Sven Guckes http://www.math.fu-berlin.de/~leitner/sed/ # Felix von Leitner http://www.dbnet.ece.ntua.gr/~george/sed/ # Yiorgos Adamopoulos http://www.student.northpark.edu/pemente/sed/ # Eric Pement
http://spacsun.rice.edu/FAQ/sed.html ftp://algos.inesc.pt/pub/users/cdua/scripts.tar.gz (sed and shell scripts)
"Handy One-Liners For Sed", compiled by Eric Pement. A large list of 1-line sed commands which can be executed from the command line.
http://sed.sourceforge.net/sed1line.txt http://www.student.northpark.edu/pemente/sed/sed1line.txt
"Handy One-Liners For Sed", translated to Portuguese
http://wmaker.lrv.ufsc.br/sed_ptBR.html
The Single UNIX Specification, Version 3 (technical man page)
http://www.opengroup.org/onlinepubs/007904975/utilities/sed.html
Getting started with sed
http://www.cs.hmc.edu/tech_docs/qref/sed.html
masm to gas converter
http://www.delorie.com/djgpp/faq/converting/asm2s-sed.html
mail2html.zip
http://www.crispen.org/src/#mail2html
sample uses of sed in batch files and scripts (Benny Pederson)
http://users.cybercity.dk/~bse26236/batutil/help/SED.HTM
dc.sed - the most complex and impressive sed script ever written. This sed script by Greg Ubben emulates the Unix dc (desk calculator), including base conversion, exponentiation, square roots, and much more.
http://sed.sourceforge.net/grabbag/scripts/dc_overview.htm
If you should find other tutorials or scripts that should be added to this document, please forward the URLs to the FAQ maintainer.
This is the end of Section 2 of the sed FAQ. |