rst 如何一次推送到多个git遥控器。如果您保留回购镜像，则非常有用。
如何一次推送到多个git遥控器。如果您保留回购镜像，则非常有用。
git-pushing-multiple.rst
Pushing to Multiple Git Repos
-----------------------------
If a project has to have multiple git repos (e.g. Bitbucket and
Github) then it's better that they remain in sync.
Usually this would involve pushing each branch to each repo in turn,
but actually Git allows pushing to multiple repos in one go.
If in doubt about what git is doing when you run these commands, just
edit ``.git/config`` (`git-config(1)`_) and see what it's put there.
Remotes
=======
Suppose your git remotes are set up like this::
git remote add github git@github.com:muccg/my-project.git
git remote add bb git@bitbucket.org:ccgmurdoch/my-project.git
The ``origin`` remote probably points to one of these URLs.
Remote Push URLs
================
To set up the push URLs do this::
git remote set-url --add --push origin git@github.com:muccg/my-project.git
git remote set-url --add --push origin git@bitbucket.org:ccgmurdoch/my-project.git
It will change the ``remote.origin.pushurl`` config entry. Now pushes
will send to both of these destinations, rather than the fetch URL.
Check it out by running::
git remote show origin
Per-branch
==========
A branch can push and pull from separate remotes. This might be useful
in rare circumstances such as maintaining a fork with customizations
to the upstream repo. If your branch follows ``github`` by default::
git branch --set-upstream-to=github next_release
(That command changed ``branch.next_release.remote``.)
Then git allows branches to have multiple ``branch.<name>.pushRemote``
entries. You must edit the ``.git/config`` file to set them.
Pull Multiple
=============
You can't pull from multiple remotes at once, but you can fetch from
all of them::
git fetch --all
Note that fetching won't update your current branch (that's why
``git-pull`` exists), so you have to merge -- fast-forward or
otherwise.
For example, this will octopus merge the branches if the remotes got
out of sync::
git merge github/next_release bb/next_release
References
==========
* `git-config(1)`_
* `git-remote(1)`_
* `git-branch(1)`_
.. _`git-config(1)`: https://www.kernel.org/pub/software/scm/git/docs/git-config.html
.. _`git-remote(1)`: https://www.kernel.org/pub/software/scm/git/docs/git-remote.html
.. _`git-branch(1)`: https://www.kernel.org/pub/software/scm/git/docs/git-branch.html
rst 使用Nginx安装Comodo PositiveSSL证书的步骤。
使用Nginx安装Comodo PositiveSSL证书的步骤。
install-comodo-ssl-cert-for-nginx.rst
Setting up a SSL Cert from Comodo
=================================
I use `Namecheap.com <http://www.namecheap.com/?aff=83780>`_ as a registrar, and they resale
SSL Certs from a number of other companies, including `Comodo <http://www.comodo.com/>`_.
These are the steps I went through to set up an SSL cert.
Purchase the cert
-----------------
Prior to purchasing a cert, you need to generate a private key, and a CSR file
(Certificate Signing Request). You'll be asked for the content of the CSR file
when ordering the certificate.
::
openssl req -new -newkey rsa:2048 -nodes -keyout example_com.key -out example_com.csr
This gives you two files:
* ``example_com.key`` -- your Private key. You'll need this later to configure ngxinx.
* ``example_com.csr`` -- Your CSR file.
Now, purchase the certificate [1]_, follow the steps on their site, and you should soon get an
email with your *PositiveSSL Certificate*. It contains a zip file with the following:
* Root CA Certificate - `AddTrustExternalCARoot.crt`
* Intermediate CA Certificate - `COMODORSAAddTrustCA.crt`
* Intermediate CA Certificate - `COMODORSADomainValidationSecureServerCA.crt`
* Your PositiveSSL Certificate - `www_example_com.crt` (or the subdomain you gave them)
Install the Commodo SSL cert
----------------------------
Combine everything for nginx [2]_:
1. Combine the above crt files into a bundle (the order matters, here)::
cat www_example_com.crt COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt AddTrustExternalCARoot.crt > ssl-bundle.crt
2. Store the bundle wherever nginx expects to find it::
mkdir -p /etc/nginx/ssl/example_com/
mv ssl-bundle.crt /etc/nginx/ssl/example_com/
3. Ensure your private key is somewhere nginx can read it, as well.::
mv example_com.key /etc/nginx/ssl/example_com/
4. Make sure your nginx config points to the right cert file and to the private
key you generated earlier::
server {
listen 443;
ssl on;
ssl_certificate /etc/nginx/ssl/example_com/ssl-bundle.crt;
ssl_certificate_key /etc/nginx/ssl/example_com/example_com.key;
# side note: only use TLS since SSLv2 and SSLv3 have had recent vulnerabilities
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
# ...
}
6. Restart nginx.
.. [1] I purchased mine through Namecheap.com.
.. [2] Based on these instructions: http://goo.gl/4zJc8
rst Markdown和reStructuredText的常见标记
Markdown和reStructuredText的常见标记
common-md-rst.md
Markdown and reStructuredText
=============================
GitHub supports several lightweight markup languages for documentation;
the most popular ones (generally, not just at GitHub) are **Markdown**
and **reStructuredText**. Markdown is sometimes considered easier to
use, and is often preferred when the purpose is simply to generate HTML.
On the other hand, reStructuredText is more extensible and powerful,
with native support (not just embedded HTML) for tables, as well as
things like automatic generation of tables of contents.
Unless you are embedding text into a programming language (especially
Python doc comments, for which purpose reStructuredText was originally
developed) you will probably choose to use Markdown for a small document
(like this Gist once was). But as you add more sections and features, you may
decide that you want the power of reStructuredText. To avoid having to
convert all of your Markdown formatting when that happens, use this Gist
as a guide to the markup syntax common to Markdown and reStructuredText.
By using only the common subset shared by these two lightweight markup
languages (largely from Setext, which inspired both of them) for your
documents, you can eliminate the need for conversion. Even if you also
use features that are specific to one or the other, using the common
subset wherever possible allows for painless cut & paste between them.
Another advantage of restricting yourself to the common subset described
here is that the resulting source is more likely to be parsed the same way
by different Markdown processors, which may handle some details like
indentation slightly differently.
If you have already used Markdown-specific syntax, or you just don't
want to limit yourself to the (admittedly very limited) intersection of
functionality in the subset, you can use **Pandoc** to convert Markdown
into reStructuredText - the online converter at
<http://johnmacfarlane.net/pandoc/try> is handy if you don't happen to
have Haskell installed. (In fact, you may find the extended Markdown
supported by Pandoc adds enough functionality that you don't need to use
reStructuredText at all - at least, once you get Haskell and Pandoc
installed.)
The Common Denominator
======================
The basics of text formatting are common to both, a blank line (it may
contain spaces or tabs, but nothing else) is used to separate paragraphs
and everything within a paragraph "flows" and is wrapped to the display
width. Be sure to avoid initial indentation (either of an entire
paragraph or just the first line) for normal text, as indentation is
significant to both Markdown and reStructuredText, but in different
ways.
When writing text, you should try to wrap source lines to 72
characters - but only when initially entering them - avoid re-wrapping
text when you edit it later, to limit the number of "insignificant
changes" in your version control system (you *are* using one, right?!).
Feel free to use two spaces after the period at the end of a sentence
(or not) if that is useful for your text editor - I'm looking at you,
Emacs - since multiple spaces are treated the same as a single space.
Similarly, multiple blank lines are treated just like one blank line;
you may want to use multiple blank lines at the end of a section.
Font Faces - Emphasis and Examples
----------------------------------
Within paragraphs, inline markup provides basic formatting to add
emphasis to particular words or phrases, most commonly by making them
*italic* or **bold** (depending on the font, italics are often rendered
with an oblique face instead). In technical writing, ``monospaced
text`` may be used to highlight program or function names, or for very
short snippets of code.
As with many types of formatting, Markdown provides multiple ways of
specifying font faces, one of which is also used by reStructuredText:
italics are marked with one asterisk (``*``) and bold face with two. There
must be whitespace or other punctuation before the leading asterisks,
and after the trailing ones; furthermore there must be no space between
the asterisks and the text being emphasized. Although Markdown supports
nesting bold and italic, reStructuredText does not (this is one of the
rare cases where Markdown is more powerful, as there is no way to
represent bold italics in reStructuredText).
Monospaced text is marked with two backquotes "``" instead of asterisks;
no bold or italic is possible within it (asterisks just represent
themselves), although in some contexts, code syntax highlighting may be
applied. Note that in monospaced text, multiple spaces are *not*
collapsed, but are preserved; however, flow and wrapping *do* occur, and
any number of spaces may be replaced by a line break. Markdown allows
monospaced text within bold or italic sections, but not vice versa -
reStructuredText allows neither. In summary, the common inline markup
is the following::
Mark *italic text* with one asterisk, **bold text** with two.
For ``monospaced text``, use two "backquotes" instead.
Mark *italic text* with one asterisk, **bold text** with two.
For ``monospaced text``, use two "backquotes" instead.
-----
(Markdown allows a single backquote to indicate monospaced text, however
double backquotes ar better, since they allow for embedded backquotes
and are required for reStructuredText.)
Escaping Markup
---------------
Sometimes you may want to use markup characters just as themselves,
without any special interpretation. Within double backquotes used to mark
monospaced text (or in code blocks as described below), markup characters
are not treated specially, but in other contexts, you must precede markup
characters with a backslash (\\) to escape them (this applies to the
backslash character as well)::
\*This\* is not *italic* - \\My Pictures\\ is a Windows folder
\*This\* is not *italic* - \\My Pictures\\ is a Windows folder
-----
While reStructuredText will treat a backslash before any non-whitespace
character as an escape, Markdown only treats backslash as an escape when
it precedes one of the following characters:
* \\ backslash
* \` backtick
* \* asterisk
* \_ underscore
* \{} curly braces
* \[] square brackets
* \() parentheses
* \# hash mark
* \+ plus sign
* \- minus sign (hyphen)
* \. dot
* \! exclamation mark
For compatibility, only use backslash before one of these characters, and
be sure to escape any backslashes that you want to appear in the output.
Code or Console Example Blocks
------------------------------
If you have example code or console text that you want to appear with
all line breaks and relative indentation preserved, in a monospaced text
block, there is no common format for Markdown and reStructuredText, but
you can combine the formatting for both of them by ending one paragraph
with a double colon ``::`` (for reStructuredText) and indenting the next
one by four or more spaces (for Markdown) to make it appear in
monospaced font without flow or word-wrapping::
A normal paragraph ending with ``::`` will flow and be word-wrapped::
If the next paragraph is indented by four or more spaces, it will be monospaced text, without flow (or even wrapping in some non-print cases.)
You can have multiple paragraphs like this, as long as they
are all indented by the same amount.
A normal paragraph ending with ``::`` will flow and be word-wrapped::
If the next paragraph is indented by four or more spaces, it will be monospaced text, without flow (or even wrapping in some non-print cases.)
You can have multiple paragraphs like this, as long as they
are all indented by the same amount.
-----
(We cheat a little bit here, Markdown does not interpret the
double-colon, but displays it as-is, whereas reStructuredText displays
just a single colon, but this is not too noticeable or annoying, as long
as you remember to use the double colon in your source.)
Line Blocks and Hard Line Breaks
--------------------------------
You may want to preserve line breaks in text blocks but don't want them
in monospaced text; common cases are verse (poetry or lyrics), street
addresses, and unadorned lists without bullets or numbering. Markdown
and reStructuredText use completely different syntax for this, but you
can combine the markup for both reStructuredText line blocks and
Markdown hard line breaks by starting each line with a vertical bar (``|``)
and a space and ending it with two spaces. For line breaks in the
source you don't want to preserve, omit the two spaces before the line
break and start the next line with one to three spaces instead. Put a
blank line before the start and after the end of every line block.
Line blocks were added to reStructuredText in Docutils version 0.3.5
and there are some reStructuredText formatters that do not support
them; notably the GitHub README markup does not display them correctly.
These line blocks can also contain inline markup (which in a code
example block might be displayed literally), but keep any markup within
each line, since emphasis starting on one line and ending on another
applies to vertical bars between them, which appear in Markdown output::
| *Yuku haru ya*
| *tori naki uo no*
| *me wa namida*
| -- **Matsuo Bashō**, The Narrow Road to Oku (*Oku no Hosomichi*),
Tokyo, 1996, p. 23 (Translation: Donald Keene)
| Spring is passing by!
| Birds are weeping and the eyes
| of fish fill with tears.
| *Yuku haru ya*
| *tori naki uo no*
| *me wa namida*
| -- **Matsuo Bashō**, The Narrow Road to Oku (*Oku no Hosomichi*),
Tokyo, 1996, p. 23 (Translation: Donald Keene)
| Spring is passing by!
| Birds are weeping and the eyes
| of fish fill with tears.
------
(Again, we cheat a bit, since the Markdown output includes the vertical
bars; but at least they make it very clear when you end a line without
the required two spaces, something that is quite easy to do as there is
usually no visual indication of whether they are there or not.)
Block Quotations
----------------
When quoting long blocks of text from another writer, it is common
(especially in the context of e-mail) to set it off from the main text
by indenting it, possibly adding a vertical quotation line along the
left margin. Markdown and reStructuredText use different syntax for
this, but you can combine their markup for block quotes by starting the
first line of a quotation with one space and a right angle bracket
(``>``), indenting all the remaining lines by one space as well (do not
add angle brackets to them).
Note that in reStructuredText, a block quotation cannot directly follow
a code example block - if it does it will be treated as part of the
example. A normal paragraph or an "empty comment" (a line with only two
periods (``..``) and blank lines before and after it) must separate
them.
Every block quotation must have a blank line before and after it; they
can use the same inline markup as ordinary paragraphs. Nested
quotations are possible by following a block quotation with another that
starts with two spaces and two right angle brackets; this allows up to
three levels of quotation (a fourth level is not possible since Markdown
treats four leading spaces as a code example). While two right angle
brackets can be adjacent, three adjacent right angle brackets are a
doctest block (a special kind of code example block) in reStructuredText
and must have spaces between them to prevent that interpretation::
> A block quotation will wrap and flow, and can have *inline*
``markup`` just like a regular paragraph. It will be indented on
the left (and possibly the right) margins, and may have a vertical
quotation line on the left.
>> With two spaces and two right angle brackets, a following block
quotation will be doubly indented, and will have a second vertical
quotation line along the left if quotation lines are generated.
> >> A third level of quotation is the maximum level possible.
..
> A block quotation will wrap and flow, and can have *inline*
``markup`` just like a regular paragraph. It will be indented on
the left (and possibly the right) margins, and may have a vertical
quotation line on the left.
>> With two spaces and two right angle brackets, a following block
quotation will be doubly indented, and will have a second vertical
quotation line along the left if quotation lines are generated.
> >> A third level of quotation is the maximum level possible.
------
(The cheat here is that the reStructuredText output includes the right
angle bracket(s) on the first line in addition to the indentation; this
is not ideal, but is generally acceptable when used for a quotation, and
not just indented text.)
Titles and Section headers
--------------------------
Both Markdown and reStructuredText allow you to structure your document
by adding header titles for sections and subsections. While they each
support a large number of levels of headers in different ways, the
common subset only has two levels: titles, formed by underlining the
text with ``==``, and subtitles, formed by underlining with ``--``. The
underlining must be on the very next line, and be at least
as long as the (sub)title::
Section Title
=============
The Common Denominator
======================
Subsection Subtitle
-------------------
Titles and Section headers
--------------------------
Note that a blank line after the underlining is optional, but a blank
line before the (sub)title is required.
Bulleted and Enumerated Lists
-----------------------------
In addition to (sub)section headers, both Markdown and reStructuredText
support itemized lists; these can be numbered (enumerated) or unnumbered
(bulleted) and the two types of lists can be nested within themselves
and each other. List items are lines starting (possibly after spaces
for indentation) with a bullet symbol (``*``, ``-``, or ``+``) for bulleted
lists, or a number and a period (``1.``) for enumerated lists; in both
cases followed by one or more spaces and then the item text. Although
reStructuredText supports other symbols for bulleted lists and
parentheses instead of period for enumerated lists, as well as ``#`` in
place of the number for auto-enumeration, Markdown only supports the
subset described above.
The spaces after the symbol or number determine the indentation needed
for additional item text on continuation lines or following paragraphs,
as well as the symbols or numbers for sub-lists. Symbol or number indentation
of all items at any nesting level must be the same (even for long
enumerated lists with two-digit numbers) but the indentation of the text
of different items need not be the same.
If a list item contains multiple paragraphs (separated by blank lines)
or sub-lists, the indentation of the item text must be at least four
spaces more than the item symbol or number; this usually requires extra
spaces between the number and period (or symbol) and the item text.
A blank line is required before the first item and after the last item
in every top-level list, but is optional between items. A blank line
is also required by reStructuredText before the first item of a
sub-list; omitting it sometimes appears to work, but only because the
sub-list is indented more than the item text. This extra indentation
may cause the item text to be treated as part of a definition list and
displayed in bold; in other cases, it causes the sub-list to be
wrapped within a block quote, causing both the left and right margins
to move inwards and creating a double-indent effect.
A sub-list without a preceding blank line can also work if there is no
item text preceding the sub-list; but this generates odd-looking
output that is confusing to human readers, with the first bullet or
number of the sub-list on the same line as the bullet or number of the
item in the enclosing list.
While Markdown does not require a blank line before a sub-list, a blank line
between items changes the inter-item spacing (typically by creating
``<p>`` paragraph tags). For consistent results, do not use blank lines
between items unless you must (for sub-lists), in which case use blank
lines between *all* the items at the same level (sub-list items do not
require the blank lines unless there are sub-sub-lists).
Markdown ignores the actual numbers given for enumerated lists and
always renumbers them starting with 1, but reStructuredText requires
that the numbers be in sequential order; the number of the first item
may or may not be preserved. For compatibility, always start enumerated
lists with 1 and number them sequentially. You should never mix
enumerated and bulleted items (or different bullet symbols) at the same
level; reStructuredText will reject it with an error (or, if there is a
blank line between them, create a new list). On the other hand,
Markdown processors will combine adjacent bulleted and enumerated lists
(using the formatting of the first list); to create separate lists it is
not enough to have a blank line, there must be a non-list paragraph
between them.
Because Markdown formatting requires additional indentation for extra
paragraphs of item text in lists, the approach for monospaced paragraphs
given above in *Code or Console Example Blocks* requires additional
indentation of at least **eight** spaces (not just four) for example
blocks in lists.
Finally, it is a *very* good idea to make sure that your document
source does not contain any tab characters, especially when working
with multiple levels of sub-lists. Configure your text editor to
expand all tabs into spaces; this will help to ensure that the initial
indentation is consistent and avoid errors if another editor
interprets the tabs differently.
The following two lists summarize and provide examples of the rules for
lists compatible with Markdown and reStructuredText::
* Mark bulleted lists with one of three symbols followed by a space:
1. asterisk (``*``)
2. hyphen (``-``)
3. plus sign (``+``)
* Mark enumerated lists with a number, period (``.``) and a space.
* The choice of symbol does not affect the output bullet style,
which is solely determined by nesting level.
Items can be continued on following lines indented at the same
level as the item text on the first line, and will flow and wrap
normally.
* The source indentation of item text can vary for different items
(but continuation lines must be indented by the same amount as
the item text that they are continuing).
Additional paragraphs of item text (after a blank line) also
require this indentation, with the extra requirement that it be
four to seven spaces more than the item symbol or number.
* These indentation requirements are the same for sub-list items
(but apply to their symbol or number, not their item text).
* Blank lines between list items are optional, avoid them.
+ If you *do* use them (for items with sub-lists or extra
paragraphs) put blank lines between *all* items at that level.
A non-list paragraph is required to separate adjacent enumerated and
bulleted lists, to keep Markdown from merging the second one into the
first (and using the first style for both).
1. Always put a blank line before the start of a list or sub-list.
2. Use the same bullet symbol for all items in a bulleted list.
3. Always start enumerated lists with 1.
4. Use sequentially increasing numbers for succeeding list items.
5. Do not mix numbers and/or different bullet symbols at one level
* (but this is okay for different levels or separate sublists).
6. Indent sub-lists by the same amount as the item text;
this must be 4-7 spaces more than the symbol or number.
1. if enumerated, always start them with 1.
+ (the same rules apply to sub-sub-lists, etcetera)
Additional non-sub-list paragraphs require the same indentation;
example blocks (after double colon ``::``) must be indented at
least eight spaces more than the symbol or number, like this::
* item text::
code block
7. Indent symbols or numbers the same amount for any one list level.
- (top-level list items should not have any leading indentation)
8. Align two-digit enumerated items by first digit, not the period.
9. Don't put leading zeros on enumerated items to align the periods
* (use spaces after period if you want to align the item text in source).
10. Make sure there are no tab characters in initial indentation.
11. Always put a blank line after the end of a (top-level) list.
* Mark bulleted lists with one of three symbols followed by a space:
1. asterisk (``*``)
2. hyphen (``-``)
3. plus sign (``+``)
* Mark enumerated lists with a number, period (``.``) and a space.
* The choice of symbol does not affect the output bullet style,
which is solely determined by nesting level.
Items can be continued on following lines indented at the same
level as the item text on the first line, and will flow and wrap
normally.
* The source indentation of item text can vary for different items
(but continuation lines must be indented by the same amount as
the item text that they are continuing).
Additional paragraphs of item text (after a blank line) also
require this indentation, with the extra requirement that it be
four to seven spaces more than the item symbol or number.
* These indentation requirements are the same for sub-list items
(but apply to their symbol or number, not their item text).
* Blank lines between list items are optional, avoid them.
+ If you *do* use them (for items with sub-lists or extra
paragraphs) put blank lines between *all* items at that level.
A non-list paragraph is required to separate adjacent enumerated and
bulleted lists, to keep Markdown from merging the second one into the
first (and using the first style for both).
1. Always put a blank line before the start of a list or sub-list.
2. Use the same bullet symbol for all items in a bulleted list.
3. Always start enumerated lists with 1.
4. Use sequentially increasing numbers for succeeding list items.
5. Do not mix numbers and/or different bullet symbols at one level
* (but this is okay for different levels or separate sublists).
6. Indent sub-lists by the same amount as the item text;
this must be 4-7 spaces more than the symbol or number.
1. if enumerated, always start them with 1.
+ (the same rules apply to sub-sub-lists, etcetera)
Additional non-sub-list paragraphs require the same indentation;
example blocks (after double colon ``::``) must be indented at
least eight spaces more than the symbol or number, like this::
* item text::
code block
7. Indent symbols or numbers the same amount for any one list level.
- (top-level list items should not have any leading indentation)
8. Align two-digit enumerated items by first digit, not the period.
9. Don't put leading zeros on enumerated items to align the periods
* (use spaces after period if you want to align the item text in source).
10. Make sure there are no tab characters in initial indentation.
11. Always put a blank line after the end of a (top-level) list.
Hyperlink URLs
--------------
Markdown and reStructuredText use different and incompatible syntax for
arbitrary text hyperlinks, but reStructuredText will generate hyperlinks
for e-mail addresses or URLs, and Markdown will do so as well if they
are enclosed in angle brackets (``<>``). Some Markdown processors do
not require the angle brackets, but there is little reason to omit them,
as they hardly affect readability, and explicitly specify whether or not
punctuation at the end of the URL is really part of the link. Even
relative URLs can be used if the protocol is explicitly specified::
The latest version of this document can be found at
<https://gist.github.com/1855764>; if you are viewing it there (via
HTTPS), you can download the Markdown/reStructuredText source at
<https:/gists/1855764/download>. You can contact the author via
e-mail at <alex.dupuy@mac.com>.
The latest version of this document can be found at
<https://gist.github.com/1855764>; if you are viewing it there (via
HTTPS), you can download the Markdown/reStructuredText source at
<https:/gists/1855764/download>. You can contact the author via
e-mail at <alex.dupuy@mac.com>.
-----
(Using the URLs directly for hyperlinks also means that even if a
Markdown processor has link generation disabled, a human reader can
always copy and paste the URL.)
Horizontal Rules (Transitions)
------------------------------
You can create a horizontal rule (a "transition" in reStructuredText
terminology) by placing four or more hyphens (``-``), asterisks (``*``),
or underscores (``_``) on a line by themselves, with blank lines before
and after and no indentation (trailing spaces are okay, but not leading
spaces). Although Markdown requires only three, and allows spaces
between them, reStructuredText requires four repeated punctuation
characters. Also, reStructuredText requires paragraphs before and after
the transition (code blocks or enumerated/bulleted list items are okay,
but section headers are not).
-----
Each of the following lines will produce a horizontal rule like the
one above::
****
______
----------
Incompatibilities
=================
There are a few cases where Markdown or reStructuredText will interpret
something as markup, although it is treated as ordinary text by the other,
and these differences can cause problems for a document that is intended
to be processed by either formatter. In these cases, the solution is
usually to add a backslash before the markup character so that it is not
treated specially by either markup processor.
(Square) Brackets
-----------------
While the appearance of square brackets - \[\] - in text is not treated
specially by reStructuredText, some Markdown processors may attempt to
format the contents as a hyperlink, even if there is no following
parenthetical URL or bracketed reference. To prevent this, put a
backslash before the opening square bracket (you can also put it before
the closing square bracket, but this is optional).
Solitary Backslash
------------------
Although Markdown only treats backslash as an escape character when it
precedes certain characters, reStructuredText treats backslash as an
escape unless it is followed by whitespace (in which case it removes the
backslash *and* the whitespace). Because of this, a single backslash
followed by whitespace, or a "C" escape like \\n, that appears in
Markdown will be "eaten" by reStructuredText. To prevent this, make sure
to always escape any backslashes that you want to appear in the output.
Not-Incompatible Extensions
===========================
Both Markdown and reStructuredText have markup that is not interpreted
by the other (either in the same or in an incompatible way), and which
is not too painful to read when rendered as ordinary text. Hyperlink
URLs (as noted above) fall into this category for some basic Markdown
implementations that do not implement URL recognition.
Tables
------
Markdown has no support for tables (one of its biggest weaknesses); to
create them requires embedded HTML (if that is even allowed). Some
variants have added table support (notably Pandoc Markdown, PHP
Markdown Extra, and GitHub Flavored Markdown \[GFM]) but the syntax is
not consistent between all of them. The reStructuredText format has
the additional advantage of being quite readable in original source
form (basic monospaced ASCII layout). If you indent reStructuredText
tables by four or more spaces (and make sure that the previous paragraph
does *not* end with a double colon ``::``) you will get a nicely
formatted table in reStructuredText and a readable ASCII table in
Markdown. There are two flavors of table markup in reStructuredText,
grid tables and simple tables. Grid tables are trickier to generate, but
more flexible, and look nicer in source format::
Make sure previous paragraph does not end with ``::``.
+-------+----------+------+
| Table Headings | Here |
+-------+----------+------+
| Sub | Headings | Too |
+=======+==========+======+
| cell | column spanning |
+ spans +----------+------+
| rows | normal | cell |
+-------+----------+------+
| multi | * cells can be |
| line | * formatted |
| cells | * paragraphs |
| too | |
+-------+-----------------+
Make sure previous paragraph does not end with ``::``.
+-------+----------+------+
| Table Headings | Here |
+-------+----------+------+
| Sub | Headings | Too |
+=======+==========+======+
| cell | column spanning |
+ spans +----------+------+
| rows | normal | cell |
+-------+----------+------+
| multi | * cells can be |
| line | * formatted |
| cells | * paragraphs |
| too | |
+-------+-----------------+
-----
A significant advantage of grid tables is that Pandoc Markdown supports
them, which is *not* the case for simple tables, for which Pandoc uses a
somewhat similar but incompatible format. However, for Pandoc to
actually process the formatting, the four space indentation of the grid
tables must be removed (to prevent monospaced code block formatting).
Simple tables are easier, but cells must be on a single line and cannot
span rows::
===== ========= =====
Table Headings Here
--------------- -----
Sub Headings Too
===== ========= =====
column spanning no
--------------- -----
cell cell row
column spanning spans
=============== =====
Note that lines between rows are optional and only needed to indicate
where cells in the previous line span columns (by omitting the space).
===== ========= =====
Table Headings Here
--------------- -----
Sub Headings Too
===== ========= =====
column spanning no
--------------- -----
cell cell row
column spanning spans
=============== =====
-----
Apart from the ability to span rows and do block formatting within cells
in a grid table, the actual table formatting is not affected by the use
of grid or simple tables, and depends only on the reStructuredText
processor and any style sheets it may use; for more visual compatibility
you may want to use the table style that most closely resembles the
output table.
Conversely, if you have a reason to prefer the "pipe tables" format
defined by PHP Markdown Extra and supported by GitHub Flavored Markdown
and the latest version (1.10) of Pandoc, you can get reasonable results by
making sure that the previous paragraph **does** end with a double colon
``::`` which will make the table display as monospaced ASCII in
reStructuredText.
Colons can be used to align columns in pipe tables::
| Tables | Are | Cool |
| ------------- |:-------------:| -----:|
| col 3 is | right-aligned | $1600 |
| col 2 is | centered | $12 |
| zebra stripes | are neat | $1 |
Colons can be used to align columns in pipe tables::
| Tables | Are | Cool |
| ------------- |:-------------:| -----:|
| col 3 is | right-aligned | $1600 |
| col 2 is | centered | $12 |
| zebra stripes | are neat | $1 |
-----
Also, just as for list indentation, it is a *very* good idea to make
sure that no tab characters are embedded in the tables; configure your
text editor to expand all tabs into spaces; this will help to ensure
that the source ASCII display in reStructuredText or Markdown is
properly aligned.
Comments
--------
There is no comment syntax for Markdown, but HTML comments can be used
with Markdown processors that allow them (raw HTML is often disabled
for security or other reasons, possibly with whitelisted tags allowed;
notably, GitHub and BitBucket README markdown disable HTML comments).
Standard Markdown (but not most processors) requires blank lines before
and after HTML blocks. Comments in reStructuredText use a different
syntax, but it is possible to create comments that are entirely
invisible in reStructuredText output, and only appear as periods in
Markdown output (unless HTML comments are disabled).
In the following comment examples, the reStructuredText comment /
directive marker ``..`` is followed by two more periods so that the
following blank line does not terminate the comment. For most Markdown
processors, you can use an `` `` entity instead of the two
additional periods to reduce the visual impact; but some Markdown
processors (notably the Python Markdown used by BitBucket README
processing) do not support entities outside of HTML blocks.
The following block is completely hidden from reStructuredText output,
and barely visible in Markdown output if HTML comments are allowed::
.. ..
<!--- Need blank line before this line (and the .. line above).
HTML comment written with 3 dashes so that Pandoc suppresses it.
Blank lines may appear anywhere in the comment.
All non-blank lines must be indented at least one space.
HTML comment close must be followed by a blank line and a line
that is not indented at all (if necessary that can be a line
with just two periods followed by another blank line).
--->
.. ..
<!--- Need blank line before this line (and the .. line above).
HTML comment written with 3 dashes so that Pandoc suppresses it.
Blank lines may appear anywhere in the comment.
All non-blank lines must be indented at least one space.
HTML comment close must be followed by a blank line and a line
that is not indented at all (if necessary that can be a line
with just two periods followed by another blank line).
--->
-----
You can also use a variation of the above to include Markdown markup
that will be entirely ignored by reStructuredText::
.. ..
<ul><li>Need blank line before this line (and .. line above).</li>
<li>Blank lines may appear anywhere in this section.</li>
<li>All non-blank lines must be indented at least one space.</li>
<li>HTML and text are displayed only in Markdown output.</li></ul>
<p>End of Markdown-only input must be followed by a blank line and
a line that is not indented at all (if necessary that can be a line
with just two dots followed by another blank line).</p>
.. ..
<ul><li>Need blank line before this line (and .. line above).</li>
<li>Blank lines may appear anywhere in this section.</li>
<li>All non-blank lines must be indented at least one space.</li>
<li>HTML and text are displayed only in Markdown output.</li></ul>
<p>End of Markdown-only input must be followed by a blank line and
a line that is not indented at all (if necessary that can be a line
with just two dots followed by another blank line).</p>
-----
You can use another variation of the above to include reStructuredText
markup that will be ignored by Markdown (except for the periods)::
.. ..
<!--- Need blank line before this line (and the .. line above).
HTML comment written with 3 dashes so that Pandoc suppresses it.
These lines not starting with .. must be indented.
HTML comment close must be followed by a blank line and a line
that is not indented at all (if necessary that can be a line
with just two periods followed by another blank line).
.. note:: This is a reStructuredText directive - the Markdown
output should be just periods
.. --->
.. ..
<!--- Need blank line before this line (and the .. line above).
HTML comment written with 3 dashes so that Pandoc suppresses it.
These lines not starting with .. must be indented.
HTML comment close must be followed by a blank line and a line
that is not indented at all (if necessary that can be a line
with just two periods followed by another blank line).
.. note:: This is a reStructuredText directive - the Markdown
output should be just periods
.. --->
-----
Note that although HTML comments are usually marked with ``<!-- -->``
you should use three dashes instead of two: ``<!--- --->`` as this is
used by Pandoc to prevent passing the comment through to the output.
Markdown Extensions
===================
Unlike reStructuredText, which is virtually identical across all its
implementations, there are a wide variety of semi-compatible Markdown
extension styles; the most popular are MultiMarkdown and Markdown Extra
(the latter implemented by PHP Markdown and Maruku, and partially by
Python Markdown and Redcarpet); Pandoc has its own set of Markdown
extensions, based on both Markdown Extra and reStructuredText; these
Markdown extensions are the most similar to reStructuredText, while the
Markdown Extra extensions have a smaller overlap, and the MultiMarkdown
extensions are only compatible with reStructuredText when they are also
identical to parts of Markdown Extra.
Definition Lists
----------------
Markdown Extra, MultiMarkdown, and Pandoc support a syntax that is
fairly compatible with the definition list syntax in reStructuredText;
by using the following format, definitions can be written that are
recognized by all of these processors. In reStructuredText, any line
that is followed immediately by an indented line is a definition term,
with the following lines at the same indentation level forming the
definition. Markdown Extra allows an optional blank line between the
term and definition lines, but requires the definition to begin with a
colon (``:``) that is not indented by more than three spaces and is
followed by a space and the definition
To be recognized as a definition list item in both reStructuredText and
Markdown extensions, only a single term is allowed, and it must be
followed immediately (with no blank line) by the definition. The
definition must begin with an (indented) colon and a space and it and
any continuation lines or additional paragraphs or definitions must all
be indented by the same amount (one to three spaces), as shown in the
following example::
term
: definition
longer term
: multi-line definition
a second line (will be subject to flow and wrapping)
a second paragraph in the definition
complex term
: first definition
: second definition
term
: definition
longer term
: multi-line definition
a second line (will be subject to flow and wrapping)
a second paragraph in the definition
complex term
: first definition
: second definition
Fancy list numbers
------------------
Although most Markdown processors only support enumerated lists with
arabic numbers followed by periods, Pandoc also supports other list
styles that are compatible with reStructuredText. In particular,
letters (``A``) as well as roman numerals (``IV``) and alternate
punctuation with parentheses ( ``(b)`` or ``xiv)`` ) are recognized, and
sequences starting with numbers other than 1 (or roman numeral I or
letter A) have the actual starting number or letter preserved in output.
README.rst
Markdown and reStructuredText
=============================
GitHub supports several lightweight markup languages for documentation;
the most popular ones (generally, not just at GitHub) are **Markdown**
and **reStructuredText**. Markdown is sometimes considered easier to
use, and is often preferred when the purpose is simply to generate HTML.
On the other hand, reStructuredText is more extensible and powerful,
with native support (not just embedded HTML) for tables, as well as
things like automatic generation of tables of contents.
Unless you are embedding text into a programming language (especially
Python doc comments, for which purpose reStructuredText was originally
developed) you will probably choose to use Markdown for a small document
(like this Gist once was). But as you add more sections and features, you may
decide that you want the power of reStructuredText. To avoid having to
convert all of your Markdown formatting when that happens, use this Gist
as a guide to the markup syntax common to Markdown and reStructuredText.
By using only the common subset shared by these two lightweight markup
languages (largely from Setext, which inspired both of them) for your
documents, you can eliminate the need for conversion. Even if you also
use features that are specific to one or the other, using the common
subset wherever possible allows for painless cut & paste between them.
Another advantage of restricting yourself to the common subset described
here is that the resulting source is more likely to be parsed the same way
by different Markdown processors, which may handle some details like
indentation slightly differently.
If you have already used Markdown-specific syntax, or you just don't
want to limit yourself to the (admittedly very limited) intersection of
functionality in the subset, you can use **Pandoc** to convert Markdown
into reStructuredText - the online converter at
<http://johnmacfarlane.net/pandoc/try> is handy if you don't happen to
have Haskell installed. (In fact, you may find the extended Markdown
supported by Pandoc adds enough functionality that you don't need to use
reStructuredText at all - at least, once you get Haskell and Pandoc
installed.)
The Common Denominator
======================
The basics of text formatting are common to both, a blank line (it may
contain spaces or tabs, but nothing else) is used to separate paragraphs
and everything within a paragraph "flows" and is wrapped to the display
width. Be sure to avoid initial indentation (either of an entire
paragraph or just the first line) for normal text, as indentation is
significant to both Markdown and reStructuredText, but in different
ways.
When writing text, you should try to wrap source lines to 72
characters - but only when initially entering them - avoid re-wrapping
text when you edit it later, to limit the number of "insignificant
changes" in your version control system (you *are* using one, right?!).
Feel free to use two spaces after the period at the end of a sentence
(or not) if that is useful for your text editor - I'm looking at you,
Emacs - since multiple spaces are treated the same as a single space.
Similarly, multiple blank lines are treated just like one blank line;
you may want to use multiple blank lines at the end of a section.
Font Faces - Emphasis and Examples
----------------------------------
Within paragraphs, inline markup provides basic formatting to add
emphasis to particular words or phrases, most commonly by making them
*italic* or **bold** (depending on the font, italics are often rendered
with an oblique face instead). In technical writing, ``monospaced
text`` may be used to highlight program or function names, or for very
short snippets of code.
As with many types of formatting, Markdown provides multiple ways of
specifying font faces, one of which is also used by reStructuredText:
italics are marked with one asterisk (``*``) and bold face with two. There
must be whitespace or other punctuation before the leading asterisks,
and after the trailing ones; furthermore there must be no space between
the asterisks and the text being emphasized. Although Markdown supports
nesting bold and italic, reStructuredText does not (this is one of the
rare cases where Markdown is more powerful, as there is no way to
represent bold italics in reStructuredText).
Monospaced text is marked with two backquotes "``" instead of asterisks;
no bold or italic is possible within it (asterisks just represent
themselves), although in some contexts, code syntax highlighting may be
applied. Note that in monospaced text, multiple spaces are *not*
collapsed, but are preserved; however, flow and wrapping *do* occur, and
any number of spaces may be replaced by a line break. Markdown allows
monospaced text within bold or italic sections, but not vice versa -
reStructuredText allows neither. In summary, the common inline markup
is the following::
Mark *italic text* with one asterisk, **bold text** with two.
For ``monospaced text``, use two "backquotes" instead.
Mark *italic text* with one asterisk, **bold text** with two.
For ``monospaced text``, use two "backquotes" instead.
-----
(Markdown allows a single backquote to indicate monospaced text, however
double backquotes ar better, since they allow for embedded backquotes
and are required for reStructuredText.)
Escaping Markup
---------------
Sometimes you may want to use markup characters just as themselves,
without any special interpretation. Within double backquotes used to mark
monospaced text (or in code blocks as described below), markup characters
are not treated specially, but in other contexts, you must precede markup
characters with a backslash (\\) to escape them (this applies to the
backslash character as well)::
\*This\* is not *italic* - \\My Pictures\\ is a Windows folder
\*This\* is not *italic* - \\My Pictures\\ is a Windows folder
-----
While reStructuredText will treat a backslash before any non-whitespace
character as an escape, Markdown only treats backslash as an escape when
it precedes one of the following characters:
* \\ backslash
* \` backtick
* \* asterisk
* \_ underscore
* \{} curly braces
* \[] square brackets
* \() parentheses
* \# hash mark
* \+ plus sign
* \- minus sign (hyphen)
* \. dot
* \! exclamation mark
For compatibility, only use backslash before one of these characters, and
be sure to escape any backslashes that you want to appear in the output.
Code or Console Example Blocks
------------------------------
If you have example code or console text that you want to appear with
all line breaks and relative indentation preserved, in a monospaced text
block, there is no common format for Markdown and reStructuredText, but
you can combine the formatting for both of them by ending one paragraph
with a double colon ``::`` (for reStructuredText) and indenting the next
one by four or more spaces (for Markdown) to make it appear in
monospaced font without flow or word-wrapping::
A normal paragraph ending with ``::`` will flow and be word-wrapped::
If the next paragraph is indented by four or more spaces, it will be monospaced text, without flow (or even wrapping in some non-print cases.)
You can have multiple paragraphs like this, as long as they
are all indented by the same amount.
A normal paragraph ending with ``::`` will flow and be word-wrapped::
If the next paragraph is indented by four or more spaces, it will be monospaced text, without flow (or even wrapping in some non-print cases.)
You can have multiple paragraphs like this, as long as they
are all indented by the same amount.
-----
(We cheat a little bit here, Markdown does not interpret the
double-colon, but displays it as-is, whereas reStructuredText displays
just a single colon, but this is not too noticeable or annoying, as long
as you remember to use the double colon in your source.)
Line Blocks and Hard Line Breaks
--------------------------------
You may want to preserve line breaks in text blocks but don't want them
in monospaced text; common cases are verse (poetry or lyrics), street
addresses, and unadorned lists without bullets or numbering. Markdown
and reStructuredText use completely different syntax for this, but you
can combine the markup for both reStructuredText line blocks and
Markdown hard line breaks by starting each line with a vertical bar (``|``)
and a space and ending it with two spaces. For line breaks in the
source you don't want to preserve, omit the two spaces before the line
break and start the next line with one to three spaces instead. Put a
blank line before the start and after the end of every line block.
Line blocks were added to reStructuredText in Docutils version 0.3.5
and there are some reStructuredText formatters that do not support
them; notably the GitHub README markup does not display them correctly.
These line blocks can also contain inline markup (which in a code
example block might be displayed literally), but keep any markup within
each line, since emphasis starting on one line and ending on another
applies to vertical bars between them, which appear in Markdown output::
| *Yuku haru ya*
| *tori naki uo no*
| *me wa namida*
| -- **Matsuo Bashō**, The Narrow Road to Oku (*Oku no Hosomichi*),
Tokyo, 1996, p. 23 (Translation: Donald Keene)
| Spring is passing by!
| Birds are weeping and the eyes
| of fish fill with tears.
| *Yuku haru ya*
| *tori naki uo no*
| *me wa namida*
| -- **Matsuo Bashō**, The Narrow Road to Oku (*Oku no Hosomichi*),
Tokyo, 1996, p. 23 (Translation: Donald Keene)
| Spring is passing by!
| Birds are weeping and the eyes
| of fish fill with tears.
------
(Again, we cheat a bit, since the Markdown output includes the vertical
bars; but at least they make it very clear when you end a line without
the required two spaces, something that is quite easy to do as there is
usually no visual indication of whether they are there or not.)
Block Quotations
----------------
When quoting long blocks of text from another writer, it is common
(especially in the context of e-mail) to set it off from the main text
by indenting it, possibly adding a vertical quotation line along the
left margin. Markdown and reStructuredText use different syntax for
this, but you can combine their markup for block quotes by starting the
first line of a quotation with one space and a right angle bracket
(``>``), indenting all the remaining lines by one space as well (do not
add angle brackets to them).
Note that in reStructuredText, a block quotation cannot directly follow
a code example block - if it does it will be treated as part of the
example. A normal paragraph or an "empty comment" (a line with only two
periods (``..``) and blank lines before and after it) must separate
them.
Every block quotation must have a blank line before and after it; they
can use the same inline markup as ordinary paragraphs. Nested
quotations are possible by following a block quotation with another that
starts with two spaces and two right angle brackets; this allows up to
three levels of quotation (a fourth level is not possible since Markdown
treats four leading spaces as a code example). While two right angle
brackets can be adjacent, three adjacent right angle brackets are a
doctest block (a special kind of code example block) in reStructuredText
and must have spaces between them to prevent that interpretation::
> A block quotation will wrap and flow, and can have *inline*
``markup`` just like a regular paragraph. It will be indented on
the left (and possibly the right) margins, and may have a vertical
quotation line on the left.
>> With two spaces and two right angle brackets, a following block
quotation will be doubly indented, and will have a second vertical
quotation line along the left if quotation lines are generated.
> >> A third level of quotation is the maximum level possible.
..
> A block quotation will wrap and flow, and can have *inline*
``markup`` just like a regular paragraph. It will be indented on
the left (and possibly the right) margins, and may have a vertical
quotation line on the left.
>> With two spaces and two right angle brackets, a following block
quotation will be doubly indented, and will have a second vertical
quotation line along the left if quotation lines are generated.
> >> A third level of quotation is the maximum level possible.
------
(The cheat here is that the reStructuredText output includes the right
angle bracket(s) on the first line in addition to the indentation; this
is not ideal, but is generally acceptable when used for a quotation, and
not just indented text.)
Titles and Section headers
--------------------------
Both Markdown and reStructuredText allow you to structure your document
by adding header titles for sections and subsections. While they each
support a large number of levels of headers in different ways, the
common subset only has two levels: titles, formed by underlining the
text with ``==``, and subtitles, formed by underlining with ``--``. The
underlining must be on the very next line, and be at least
as long as the (sub)title::
Section Title
=============
The Common Denominator
======================
Subsection Subtitle
-------------------
Titles and Section headers
--------------------------
Note that a blank line after the underlining is optional, but a blank
line before the (sub)title is required.
Bulleted and Enumerated Lists
-----------------------------
In addition to (sub)section headers, both Markdown and reStructuredText
support itemized lists; these can be numbered (enumerated) or unnumbered
(bulleted) and the two types of lists can be nested within themselves
and each other. List items are lines starting (possibly after spaces
for indentation) with a bullet symbol (``*``, ``-``, or ``+``) for bulleted
lists, or a number and a period (``1.``) for enumerated lists; in both
cases followed by one or more spaces and then the item text. Although
reStructuredText supports other symbols for bulleted lists and
parentheses instead of period for enumerated lists, as well as ``#`` in
place of the number for auto-enumeration, Markdown only supports the
subset described above.
The spaces after the symbol or number determine the indentation needed
for additional item text on continuation lines or following paragraphs,
as well as the symbols or numbers for sub-lists. Symbol or number indentation
of all items at any nesting level must be the same (even for long
enumerated lists with two-digit numbers) but the indentation of the text
of different items need not be the same.
If a list item contains multiple paragraphs (separated by blank lines)
or sub-lists, the indentation of the item text must be at least four
spaces more than the item symbol or number; this usually requires extra
spaces between the number and period (or symbol) and the item text.
A blank line is required before the first item and after the last item
in every top-level list, but is optional between items. A blank line
is also required by reStructuredText before the first item of a
sub-list; omitting it sometimes appears to work, but only because the
sub-list is indented more than the item text. This extra indentation
may cause the item text to be treated as part of a definition list and
displayed in bold; in other cases, it causes the sub-list to be
wrapped within a block quote, causing both the left and right margins
to move inwards and creating a double-indent effect.
A sub-list without a preceding blank line can also work if there is no
item text preceding the sub-list; but this generates odd-looking
output that is confusing to human readers, with the first bullet or
number of the sub-list on the same line as the bullet or number of the
item in the enclosing list.
While Markdown does not require a blank line before a sub-list, a blank line
between items changes the inter-item spacing (typically by creating
``<p>`` paragraph tags). For consistent results, do not use blank lines
between items unless you must (for sub-lists), in which case use blank
lines between *all* the items at the same level (sub-list items do not
require the blank lines unless there are sub-sub-lists).
Markdown ignores the actual numbers given for enumerated lists and
always renumbers them starting with 1, but reStructuredText requires
that the numbers be in sequential order; the number of the first item
may or may not be preserved. For compatibility, always start enumerated
lists with 1 and number them sequentially. You should never mix
enumerated and bulleted items (or different bullet symbols) at the same
level; reStructuredText will reject it with an error (or, if there is a
blank line between them, create a new list). On the other hand,
Markdown processors will combine adjacent bulleted and enumerated lists
(using the formatting of the first list); to create separate lists it is
not enough to have a blank line, there must be a non-list paragraph
between them.
Because Markdown formatting requires additional indentation for extra
paragraphs of item text in lists, the approach for monospaced paragraphs
given above in *Code or Console Example Blocks* requires additional
indentation of at least **eight** spaces (not just four) for example
blocks in lists.
Finally, it is a *very* good idea to make sure that your document
source does not contain any tab characters, especially when working
with multiple levels of sub-lists. Configure your text editor to
expand all tabs into spaces; this will help to ensure that the initial
indentation is consistent and avoid errors if another editor
interprets the tabs differently.
The following two lists summarize and provide examples of the rules for
lists compatible with Markdown and reStructuredText::
* Mark bulleted lists with one of three symbols followed by a space:
1. asterisk (``*``)
2. hyphen (``-``)
3. plus sign (``+``)
* Mark enumerated lists with a number, period (``.``) and a space.
* The choice of symbol does not affect the output bullet style,
which is solely determined by nesting level.
Items can be continued on following lines indented at the same
level as the item text on the first line, and will flow and wrap
normally.
* The source indentation of item text can vary for different items
(but continuation lines must be indented by the same amount as
the item text that they are continuing).
Additional paragraphs of item text (after a blank line) also
require this indentation, with the extra requirement that it be
four to seven spaces more than the item symbol or number.
* These indentation requirements are the same for sub-list items
(but apply to their symbol or number, not their item text).
* Blank lines between list items are optional, avoid them.
+ If you *do* use them (for items with sub-lists or extra
paragraphs) put blank lines between *all* items at that level.
A non-list paragraph is required to separate adjacent enumerated and
bulleted lists, to keep Markdown from merging the second one into the
first (and using the first style for both).
1. Always put a blank line before the start of a list or sub-list.
2. Use the same bullet symbol for all items in a bulleted list.
3. Always start enumerated lists with 1.
4. Use sequentially increasing numbers for succeeding list items.
5. Do not mix numbers and/or different bullet symbols at one level
* (but this is okay for different levels or separate sublists).
6. Indent sub-lists by the same amount as the item text;
this must be 4-7 spaces more than the symbol or number.
1. if enumerated, always start them with 1.
+ (the same rules apply to sub-sub-lists, etcetera)
Additional non-sub-list paragraphs require the same indentation;
example blocks (after double colon ``::``) must be indented at
least eight spaces more than the symbol or number, like this::
* item text::
code block
7. Indent symbols or numbers the same amount for any one list level.
- (top-level list items should not have any leading indentation)
8. Align two-digit enumerated items by first digit, not the period.
9. Don't put leading zeros on enumerated items to align the periods
* (use spaces after period if you want to align the item text in source).
10. Make sure there are no tab characters in initial indentation.
11. Always put a blank line after the end of a (top-level) list.
* Mark bulleted lists with one of three symbols followed by a space:
1. asterisk (``*``)
2. hyphen (``-``)
3. plus sign (``+``)
* Mark enumerated lists with a number, period (``.``) and a space.
* The choice of symbol does not affect the output bullet style,
which is solely determined by nesting level.
Items can be continued on following lines indented at the same
level as the item text on the first line, and will flow and wrap
normally.
* The source indentation of item text can vary for different items
(but continuation lines must be indented by the same amount as
the item text that they are continuing).
Additional paragraphs of item text (after a blank line) also
require this indentation, with the extra requirement that it be
four to seven spaces more than the item symbol or number.
* These indentation requirements are the same for sub-list items
(but apply to their symbol or number, not their item text).
* Blank lines between list items are optional, avoid them.
+ If you *do* use them (for items with sub-lists or extra
paragraphs) put blank lines between *all* items at that level.
A non-list paragraph is required to separate adjacent enumerated and
bulleted lists, to keep Markdown from merging the second one into the
first (and using the first style for both).
1. Always put a blank line before the start of a list or sub-list.
2. Use the same bullet symbol for all items in a bulleted list.
3. Always start enumerated lists with 1.
4. Use sequentially increasing numbers for succeeding list items.
5. Do not mix numbers and/or different bullet symbols at one level
* (but this is okay for different levels or separate sublists).
6. Indent sub-lists by the same amount as the item text;
this must be 4-7 spaces more than the symbol or number.
1. if enumerated, always start them with 1.
+ (the same rules apply to sub-sub-lists, etcetera)
Additional non-sub-list paragraphs require the same indentation;
example blocks (after double colon ``::``) must be indented at
least eight spaces more than the symbol or number, like this::
* item text::
code block
7. Indent symbols or numbers the same amount for any one list level.
- (top-level list items should not have any leading indentation)
8. Align two-digit enumerated items by first digit, not the period.
9. Don't put leading zeros on enumerated items to align the periods
* (use spaces after period if you want to align the item text in source).
10. Make sure there are no tab characters in initial indentation.
11. Always put a blank line after the end of a (top-level) list.
Hyperlink URLs
--------------
Markdown and reStructuredText use different and incompatible syntax for
arbitrary text hyperlinks, but reStructuredText will generate hyperlinks
for e-mail addresses or URLs, and Markdown will do so as well if they
are enclosed in angle brackets (``<>``). Some Markdown processors do
not require the angle brackets, but there is little reason to omit them,
as they hardly affect readability, and explicitly specify whether or not
punctuation at the end of the URL is really part of the link. Even
relative URLs can be used if the protocol is explicitly specified::
The latest version of this document can be found at
<https://gist.github.com/1855764>; if you are viewing it there (via
HTTPS), you can download the Markdown/reStructuredText source at
<https:/gists/1855764/download>. You can contact the author via
e-mail at <alex.dupuy@mac.com>.
The latest version of this document can be found at
<https://gist.github.com/1855764>; if you are viewing it there (via
HTTPS), you can download the Markdown/reStructuredText source at
<https:/gists/1855764/download>. You can contact the author via
e-mail at <alex.dupuy@mac.com>.
-----
(Using the URLs directly for hyperlinks also means that even if a
Markdown processor has link generation disabled, a human reader can
always copy and paste the URL.)
Horizontal Rules (Transitions)
------------------------------
You can create a horizontal rule (a "transition" in reStructuredText
terminology) by placing four or more hyphens (``-``), asterisks (``*``),
or underscores (``_``) on a line by themselves, with blank lines before
and after and no indentation (trailing spaces are okay, but not leading
spaces). Although Markdown requires only three, and allows spaces
between them, reStructuredText requires four repeated punctuation
characters. Also, reStructuredText requires paragraphs before and after
the transition (code blocks or enumerated/bulleted list items are okay,
but section headers are not).
-----
Each of the following lines will produce a horizontal rule like the
one above::
****
______
----------
Incompatibilities
=================
There are a few cases where Markdown or reStructuredText will interpret
something as markup, although it is treated as ordinary text by the other,
and these differences can cause problems for a document that is intended
to be processed by either formatter. In these cases, the solution is
usually to add a backslash before the markup character so that it is not
treated specially by either markup processor.
(Square) Brackets
-----------------
While the appearance of square brackets - \[\] - in text is not treated
specially by reStructuredText, some Markdown processors may attempt to
format the contents as a hyperlink, even if there is no following
parenthetical URL or bracketed reference. To prevent this, put a
backslash before the opening square bracket (you can also put it before
the closing square bracket, but this is optional).
Solitary Backslash
------------------
Although Markdown only treats backslash as an escape character when it
precedes certain characters, reStructuredText treats backslash as an
escape unless it is followed by whitespace (in which case it removes the
backslash *and* the whitespace). Because of this, a single backslash
followed by whitespace, or a "C" escape like \\n, that appears in
Markdown will be "eaten" by reStructuredText. To prevent this, make sure
to always escape any backslashes that you want to appear in the output.
Not-Incompatible Extensions
===========================
Both Markdown and reStructuredText have markup that is not interpreted
by the other (either in the same or in an incompatible way), and which
is not too painful to read when rendered as ordinary text. Hyperlink
URLs (as noted above) fall into this category for some basic Markdown
implementations that do not implement URL recognition.
Tables
------
Markdown has no support for tables (one of its biggest weaknesses); to
create them requires embedded HTML (if that is even allowed). Some
variants have added table support (notably Pandoc Markdown, PHP
Markdown Extra, and GitHub Flavored Markdown \[GFM]) but the syntax is
not consistent between all of them. The reStructuredText format has
the additional advantage of being quite readable in original source
form (basic monospaced ASCII layout). If you indent reStructuredText
tables by four or more spaces (and make sure that the previous paragraph
does *not* end with a double colon ``::``) you will get a nicely
formatted table in reStructuredText and a readable ASCII table in
Markdown. There are two flavors of table markup in reStructuredText,
grid tables and simple tables. Grid tables are trickier to generate, but
more flexible, and look nicer in source format::
Make sure previous paragraph does not end with ``::``.
+-------+----------+------+
| Table Headings | Here |
+-------+----------+------+
| Sub | Headings | Too |
+=======+==========+======+
| cell | column spanning |
+ spans +----------+------+
| rows | normal | cell |
+-------+----------+------+
| multi | * cells can be |
| line | * formatted |
| cells | * paragraphs |
| too | |
+-------+-----------------+
Make sure previous paragraph does not end with ``::``.
+-------+----------+------+
| Table Headings | Here |
+-------+----------+------+
| Sub | Headings | Too |
+=======+==========+======+
| cell | column spanning |
+ spans +----------+------+
| rows | normal | cell |
+-------+----------+------+
| multi | * cells can be |
| line | * formatted |
| cells | * paragraphs |
| too | |
+-------+-----------------+
-----
A significant advantage of grid tables is that Pandoc Markdown supports
them, which is *not* the case for simple tables, for which Pandoc uses a
somewhat similar but incompatible format. However, for Pandoc to
actually process the formatting, the four space indentation of the grid
tables must be removed (to prevent monospaced code block formatting).
Simple tables are easier, but cells must be on a single line and cannot
span rows::
===== ========= =====
Table Headings Here
--------------- -----
Sub Headings Too
===== ========= =====
column spanning no
--------------- -----
cell cell row
column spanning spans
=============== =====
Note that lines between rows are optional and only needed to indicate
where cells in the previous line span columns (by omitting the space).
===== ========= =====
Table Headings Here
--------------- -----
Sub Headings Too
===== ========= =====
column spanning no
--------------- -----
cell cell row
column spanning spans
=============== =====
-----
Apart from the ability to span rows and do block formatting within cells
in a grid table, the actual table formatting is not affected by the use
of grid or simple tables, and depends only on the reStructuredText
processor and any style sheets it may use; for more visual compatibility
you may want to use the table style that most closely resembles the
output table.
Conversely, if you have a reason to prefer the "pipe tables" format
defined by PHP Markdown Extra and supported by GitHub Flavored Markdown
and the latest version (1.10) of Pandoc, you can get reasonable results by
making sure that the previous paragraph **does** end with a double colon
``::`` which will make the table display as monospaced ASCII in
reStructuredText.
Colons can be used to align columns in pipe tables::
| Tables | Are | Cool |
| ------------- |:-------------:| -----:|
| col 3 is | right-aligned | $1600 |
| col 2 is | centered | $12 |
| zebra stripes | are neat | $1 |
Colons can be used to align columns in pipe tables::
| Tables | Are | Cool |
| ------------- |:-------------:| -----:|
| col 3 is | right-aligned | $1600 |
| col 2 is | centered | $12 |
| zebra stripes | are neat | $1 |
-----
Also, just as for list indentation, it is a *very* good idea to make
sure that no tab characters are embedded in the tables; configure your
text editor to expand all tabs into spaces; this will help to ensure
that the source ASCII display in reStructuredText or Markdown is
properly aligned.
Comments
--------
There is no comment syntax for Markdown, but HTML comments can be used
with Markdown processors that allow them (raw HTML is often disabled
for security or other reasons, possibly with whitelisted tags allowed;
notably, GitHub and BitBucket README markdown disable HTML comments).
Standard Markdown (but not most processors) requires blank lines before
and after HTML blocks. Comments in reStructuredText use a different
syntax, but it is possible to create comments that are entirely
invisible in reStructuredText output, and only appear as periods in
Markdown output (unless HTML comments are disabled).
In the following comment examples, the reStructuredText comment /
directive marker ``..`` is followed by two more periods so that the
following blank line does not terminate the comment. For most Markdown
processors, you can use an `` `` entity instead of the two
additional periods to reduce the visual impact; but some Markdown
processors (notably the Python Markdown used by BitBucket README
processing) do not support entities outside of HTML blocks.
The following block is completely hidden from reStructuredText output,
and barely visible in Markdown output if HTML comments are allowed::
.. ..
<!--- Need blank line before this line (and the .. line above).
HTML comment written with 3 dashes so that Pandoc suppresses it.
Blank lines may appear anywhere in the comment.
All non-blank lines must be indented at least one space.
HTML comment close must be followed by a blank line and a line
that is not indented at all (if necessary that can be a line
with just two periods followed by another blank line).
--->
.. ..
<!--- Need blank line before this line (and the .. line above).
HTML comment written with 3 dashes so that Pandoc suppresses it.
Blank lines may appear anywhere in the comment.
All non-blank lines must be indented at least one space.
HTML comment close must be followed by a blank line and a line
that is not indented at all (if necessary that can be a line
with just two periods followed by another blank line).
--->
-----
You can also use a variation of the above to include Markdown markup
that will be entirely ignored by reStructuredText::
.. ..
<ul><li>Need blank line before this line (and .. line above).</li>
<li>Blank lines may appear anywhere in this section.</li>
<li>All non-blank lines must be indented at least one space.</li>
<li>HTML and text are displayed only in Markdown output.</li></ul>
<p>End of Markdown-only input must be followed by a blank line and
a line that is not indented at all (if necessary that can be a line
with just two dots followed by another blank line).</p>
.. ..
<ul><li>Need blank line before this line (and .. line above).</li>
<li>Blank lines may appear anywhere in this section.</li>
<li>All non-blank lines must be indented at least one space.</li>
<li>HTML and text are displayed only in Markdown output.</li></ul>
<p>End of Markdown-only input must be followed by a blank line and
a line that is not indented at all (if necessary that can be a line
with just two dots followed by another blank line).</p>
-----
You can use another variation of the above to include reStructuredText
markup that will be ignored by Markdown (except for the periods)::
.. ..
<!--- Need blank line before this line (and the .. line above).
HTML comment written with 3 dashes so that Pandoc suppresses it.
These lines not starting with .. must be indented.
HTML comment close must be followed by a blank line and a line
that is not indented at all (if necessary that can be a line
with just two periods followed by another blank line).
.. note:: This is a reStructuredText directive - the Markdown
output should be just periods
.. --->
.. ..
<!--- Need blank line before this line (and the .. line above).
HTML comment written with 3 dashes so that Pandoc suppresses it.
These lines not starting with .. must be indented.
HTML comment close must be followed by a blank line and a line
that is not indented at all (if necessary that can be a line
with just two periods followed by another blank line).
.. note:: This is a reStructuredText directive - the Markdown
output should be just periods
.. --->
-----
Note that although HTML comments are usually marked with ``<!-- -->``
you should use three dashes instead of two: ``<!--- --->`` as this is
used by Pandoc to prevent passing the comment through to the output.
Markdown Extensions
===================
Unlike reStructuredText, which is virtually identical across all its
implementations, there are a wide variety of semi-compatible Markdown
extension styles; the most popular are MultiMarkdown and Markdown Extra
(the latter implemented by PHP Markdown and Maruku, and partially by
Python Markdown and Redcarpet); Pandoc has its own set of Markdown
extensions, based on both Markdown Extra and reStructuredText; these
Markdown extensions are the most similar to reStructuredText, while the
Markdown Extra extensions have a smaller overlap, and the MultiMarkdown
extensions are only compatible with reStructuredText when they are also
identical to parts of Markdown Extra.
Definition Lists
----------------
Markdown Extra, MultiMarkdown, and Pandoc support a syntax that is
fairly compatible with the definition list syntax in reStructuredText;
by using the following format, definitions can be written that are
recognized by all of these processors. In reStructuredText, any line
that is followed immediately by an indented line is a definition term,
with the following lines at the same indentation level forming the
definition. Markdown Extra allows an optional blank line between the
term and definition lines, but requires the definition to begin with a
colon (``:``) that is not indented by more than three spaces and is
followed by a space and the definition
To be recognized as a definition list item in both reStructuredText and
Markdown extensions, only a single term is allowed, and it must be
followed immediately (with no blank line) by the definition. The
definition must begin with an (indented) colon and a space and it and
any continuation lines or additional paragraphs or definitions must all
be indented by the same amount (one to three spaces), as shown in the
following example::
term
: definition
longer term
: multi-line definition
a second line (will be subject to flow and wrapping)
a second paragraph in the definition
complex term
: first definition
: second definition
term
: definition
longer term
: multi-line definition
a second line (will be subject to flow and wrapping)
a second paragraph in the definition
complex term
: first definition
: second definition
Fancy list numbers
------------------
Although most Markdown processors only support enumerated lists with
arabic numbers followed by periods, Pandoc also supports other list
styles that are compatible with reStructuredText. In particular,
letters (``A``) as well as roman numerals (``IV``) and alternate
punctuation with parentheses ( ``(b)`` or ``xiv)`` ) are recognized, and
sequences starting with numbers other than 1 (or roman numeral I or
letter A) have the actual starting number or letter preserved in output.
rst outbrain-database.rst
outbrain-database.rst
.. image:: https://img.shields.io/badge/People-Zi_Ran_Feng-red.svg
.. image:: https://img.shields.io/badge/People-Yu_Chen_Li-red.svg
.. image:: https://img.shields.io/badge/language-Python34-brightgreen.svg
.. image:: https://img.shields.io/badge/database-sqlite-brightgreen.svg
Define the table
----------------
Because each document may have multiple category, topic and entity. If we only use the highest confident label, then the complexity can be greatly reduced. After that, document vs category, topic and entity would be many-to-one relationship.
.. code-block:: python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sqlalchemy
from sqlalchemy import MetaData, Table, Column, ForeignKey, Index
from sqlalchemy import String, Integer, Float, DateTime
from outbrain.config import db_file_path
metadata = MetaData()
#--- Atomic object ---
t_document = Table("document", metadata,
Column("document_id", Integer, primary_key=True),
Column("source_id", Integer),
Column("publisher_id", Integer),
Column("publish_time", DateTime),
# category, topic, entity只选择最大confidence_level的那个使用
Column("category_id", Integer),
Column("category_confidence", Float),
Column("topic_id", Integer),
Column("topic_confidence", Float),
Column("entity_id", Integer),
Column("entity_confidence", Float),
)
t_ad = Table("ad", metadata,
Column("ad_id", Integer, primary_key=True),
Column("document_id", Integer),
Column("campaign_id", Integer),
Column("advertiser_id", Integer),
)
#--- Association ---
t_ad_and_document = Table("ad_and_document", metadata,
Column("document_id", Integer, primary_key=True, ForeignKey("document.document_id")),
Column("ad_id", Integer, primary_key=True, ForeignKey("ad.ad_id")),
Column("clicked", Integer),
)
engine = sqlalchemy.create_engine("sqlite:///%s" % db_file_path)
metadata.create_all(engine)
Feed the data
-------------
Because ``ad_id`` in ``clicks_train`` and ``clicks_test`` is subset of ``promoted_content``, so we only need to focus on ``promoted_content``. This script feed ad_id and it's metadata to database:
.. code-block:: python
def feed_ad():
"""
**中文文档**
建立ad_id的主表
"""
logger.info("Run feed_ad() ...")
logger.info("read promoted_content ...", 1)
p = Path(config.data_dir, "promoted_content.csv", nrows=nrows)
df = pd.read_csv(p.abspath)
logger.info("write to database ...", 1)
df.to_sql(database.t_ad.name, engine, index=False, if_exists="replace")
feed_ad()
Document table is little tricky, first we have to choose highest confident one for ``category``, ``topic`` and ``entity``. And then join them by ``document_id``.
.. code-block:: python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from datetime import datetime
import numpy as np, pandas as pd
from pathlib_mate import Path
from sfm import sqlalchemy_mate, timer
from sqlalchemy import select, func, distinct
from outbrain import config, database
from outbrain.database import engine, metadata
from outbrain.logger import logger
from outbrain.util.iotool import csv_tuple_iterator
nrows = None
#--- Categories, Topics, Entities ---
def pick_highest_confidence(filename):
"""对CSV文件进行处理, 选择confidence_level最高的那个作为document相对应的属性。
"""
logger.info("run pick_highest_confidence(%s) ..." % filename)
p = Path(config.data_dir, filename)
df_raw = pd.read_csv(p.abspath, nrows=nrows)
label_name = df_raw.columns[1]
data = dict()
counter = 0
for document_id, label, confidence_level in zip(
*(array for col, array in df_raw.iteritems()) ):
counter += 1
try:
if confidence_level > data[document_id]["confidence_level"]:
data[document_id][label_name] = label
data[document_id]["conf"] = confidence_level
except:
data[document_id] = {
"document_id": document_id,
label_name: label,
"confidence_level": confidence_level,
}
df = pd.DataFrame(list(data.values()))
return df
def merge_category_topic_entity():
"""将按照confidence_level排序后的三个表进行join, 并把join的结果写入csv。
"""
logger.info("run merge_category_topic_entity() ...")
df_cate = pick_highest_confidence("documents_categories.csv")
df_topic = pick_highest_confidence("documents_topics.csv")
df_entity = pick_highest_confidence("documents_entities.csv")
logger.info("outer join dataframe ...", 1)
df = df_cate.merge(df_topic, how="outer", on="document_id").\
merge(df_entity, how="outer", on="document_id")
df = df[["document_id",
"category_id", "confidence_level_x",
"topic_id", "confidence_level_y",
"entity_id", "confidence_level",]]
df.columns = ["document_id",
"category_id", "category_confidence",
"topic_id", "topic_confidence",
"entity_id", "entity_confidence",]
logger.info("write to csv ...", 1)
df.to_csv(Path(config.temp_data_dir, "documents_labels.csv").abspath, index=False)
# merge_category_topic_entity()
Then we could merge to ``documents_meta`` and feed into database.
.. code-block:: python
def feed_documents():
"""
**中文文档**
建立document_id的主表
"""
logger.info("read documents_meta ...", 1)
p = Path(config.data_dir, "documents_meta.csv")
df_meta = pd.read_csv(p.abspath, parse_dates=["publish_time",], nrows=nrows)
df_meta = df_meta.dropna(subset=["source_id"], inplace=False)
logger.info("read documents_labels ...", 1)
p = Path(config.temp_data_dir, "documents_labels.csv")
df_label = pd.read_csv(p.abspath, nrows=nrows)
logger.info("outer join ...", 1)
df = df_meta.merge(df_label, how="outer", on="document_id")
logger.info("write to database ...", 1)
df.to_sql(database.t_document.name, engine, index=False, if_exists="replace")
feed_documents()
rst outbrain-数据探索-report.rst
outbrain-data-explore-report.rst
.. image:: https://img.shields.io/badge/People-Zi_Ran_Feng-red.svg
.. image:: https://img.shields.io/badge/People-Yu_Chen_Li-red.svg
.. image:: https://img.shields.io/badge/language-Python34-brightgreen.svg
.. image:: https://img.shields.io/badge/database-sqlite-brightgreen.svg
Outbrain data explore report
============================
- `About the data <https://www.kaggle.com/c/outbrain-click-prediction/data>`_.
- `github <https://github.com/MacHu-GWU/outbrain-project>`_, currently private.
Understand the data
-------------------
- **scale**: find how many rows, how much disk space it cost.
- **primary_key**: find those primary objects, like document, advertisement, event, user, etc, ...
- **cardinality**: find unique number of values for each data points.
- **relationship**: find relationship between each pair of data points, could be one-to-one, one-to-many, many-to-many
- **data quality**: find how many missing values
We simple read the first 1000 rows and count lines for each csv file, and write everything into a single excel file. Now browse data is easy.
.. code-block:: python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import numpy as np, pandas as pd
from pathlib_mate import Path
from sfm import lines_count
from outbrain import config, database
from outbrain.database import engine, metadata
from outbrain.logger import logger
def merge_everything_to_schema():
"""
**中文文档**
汇总所有csv文件的Schema。
每个文件读取前1000行，以及统计总行数。然后将多个文件的信息汇总到一个
Excel表中, 以供方便观察Schema。
"""
logger.info("Run merge_everything_to_schema()")
writer = pd.ExcelWriter("Schema.xlsx")
# select all .csv file
for p in Path(config.data_dir).select_by_ext(".csv"):
logger.info("process %s ..." % p.basename, 1)
lines = lines_count.count_lines(p.abspath) - 1 # count lines
sheet_name = p.fname + "_%s" % lines
df = pd.read_csv(p.abspath, nrows=1000) # fetch first 1000 rows
df.to_excel(writer, sheet_name, index=False) # create a spreadsheet
writer.save() # write to excel
merge_everything_to_schema()
Explore the data
----------------
We noticed that the ``document_id`` is associated with ``ad_id`` and ``click_status`` via ``display_id`` in ``events.csv``. And each document has ``category``, ``topic``, ``entity`` and other ``meta_data``.
We believe the relationship is very important, and we also want to take advantage from database to boost our data-selection. Here we have some code to find out.
.. code-block:: python
def investigate_documents_categories_csv():
"""Find out the document_id vs category_id is many-to-many.
**中文文档**
检查document_id和category_id的关系:
结论: 多对多的关系。
"""
p = Path("raw_data", "documents_categories.csv")
df = pd.read_csv(p.abspath, usecols=["document_id", "category_id"])
n_rows = df.shape[0]
n_document_id = len(df["document_id"].unique())
n_category_id = len(df["category_id"].unique())
print(n_rows, n_document_id, n_category_id) # 5481475 2828649 97
investigate_documents_categories_csv()
``ad_id`` exists in ``clicks_train.csv``, ``clicks_test.csv`` and ``promoted_content.csv``, and ``promoted_content.csv`` provides rich meta data. Is meta data is available for all ``ad_id`` in train and test data? The script tells me "YES"!
.. code-block:: python
def investigate_ad_id():
"""
**中文文档**
由于在clicks_train, clicks_test和promoted_content三个文件中都有ad_id。
promoted_content中的ad_id是unique的, 而另外两个文件中的不是。所以我们要检查
clicks_train, clicks_test两个文件中的ad_id的集合是否有promoted_content中
没有的元素。
结论: promoted_content中的ad_id就是所有的ad_id。
"""
p = Path(config.data_dir, "clicks_train.csv")
df_train = pd.read_csv(p.abspath, usecols=[1,])
p = Path(config.data_dir, "clicks_test.csv")
df_test = pd.read_csv(p.abspath, usecols=[1,])
p = Path(config.data_dir, "promoted_content.csv")
df_promoted = pd.read_csv(p.abspath, usecols=[0,])
s = set.union(set(df_train["ad_id"]), set(df_test["ad_id"])).\
difference(set(df_promoted["ad_id"]))
print(len(s)) # 0
investigate_ad_id()
Baseline Model
--------------
Since the clicks_trains provides the naive probability that an ``ad_id`` will be clicked. How about for any ``display_id``, we order the ``ad_id`` by its probability? Let's have a try:
.. code-block:: python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import numpy as np, pandas as pd
from pathlib_mate import Path
from sfm.timer import Timer
from sfm.pandas_mate import IOTool
from outbrain import config, database
from outbrain.logger import logger
def naive_guess_based_on_probability():
"""
Runtime 5 minutes.
"""
table = dict()
chunksize = 1000000
with Timer():
logger.info("calculate probability ...")
p = Path(config.data_dir, "clicks_train.csv")
for _, ad_id, click in IOTool.csv_tuple_iterator(p.abspath, chunksize):
if click:
try:
table[ad_id]["all"] += 1
table[ad_id]["clicked"] += 1
except:
table[ad_id] = {"all": 1, "clicked": 1}
else:
try:
table[ad_id]["all"] += 1
except:
table[ad_id] = {"all": 1, "clicked": 0}
for _, d in table.items():
d["prob"] = d["clicked"] * 1.0 / d["all"]
with Timer():
logger.info("group by display_id ...")
display = dict()
p = Path(config.data_dir, "clicks_test.csv")
for display_id, ad_id in IOTool.csv_tuple_iterator(p.abspath, chunksize):
try:
display[display_id].append(ad_id)
except:
display[display_id] = [ad_id,]
with Timer():
logger.info("choose best ad_id ...")
result = list()
for display_id, ad_id_list in display.items():
prob_table = [(ad_id, table.get(ad_id, {"prob": 0.0})["prob"]) for ad_id in ad_id_list]
best_ad_id, best_prob = list(sorted(prob_table, key=lambda x: x[1], reverse=-1))[0]
result.append((
display_id,
best_ad_id,
best_prob,
", ".join([str(ad_id) for ad_id in ad_id_list])
))
df = pd.DataFrame(result, columns=["display_id", "best_ad_id", "best_prob", "all_ad_id"])
with Timer():
logger.info("export result ...")
df.to_csv("result.csv", index=False)
naive_guess_based_on_probability()rst rubi_feeder_remote.rst
rubi_feeder_remote.rst
http://www.instructables.com/id/Internet-Enabled-Raspberry-Pi-Pet-Feeder/
this sucks (Not recommended for dogs over 50 pounds > 23 kg)
http://www.petnet.io/
https://www.youtube.com/watch?v=qIFSuc9BW8E
http://familytechzone.com/2016/02/petnet-ios-automatic-pet-feeder/rst 使用Nginx安装Comodo PositiveSSL证书的步骤。
使用Nginx安装Comodo PositiveSSL证书的步骤。
install-comodo-ssl-cert-for-nginx.rst
Setting up a SSL Cert from Comodo
=================================
I use `Namecheap.com <http://www.namecheap.com/?aff=83780>`_ as a registrar, and they resale
SSL Certs from a number of other companies, including `Comodo <http://www.comodo.com/>`_.
These are the steps I went through to set up an SSL cert.
Purchase the cert
-----------------
Prior to purchasing a cert, you need to generate a private key, and a CSR file
(Certificate Signing Request). You'll be asked for the content of the CSR file
when ordering the certificate.
::
openssl req -new -newkey rsa:2048 -nodes -keyout example_com.key -out example_com.csr
This gives you two files:
* ``example_com.key`` -- your Private key. You'll need this later to configure ngxinx.
* ``example_com.csr`` -- Your CSR file.
Now, purchase the certificate [1]_, follow the steps on their site, and you should soon get an
email with your *PositiveSSL Certificate*. It contains a zip file with the following:
* Root CA Certificate - `AddTrustExternalCARoot.crt`
* Intermediate CA Certificate - `COMODORSAAddTrustCA.crt`
* Intermediate CA Certificate - `COMODORSADomainValidationSecureServerCA.crt`
* Your PositiveSSL Certificate - `www_example_com.crt` (or the subdomain you gave them)
Install the Commodo SSL cert
----------------------------
Combine everything for nginx [2]_:
1. Combine the above crt files into a bundle (the order matters, here)::
cat www_example_com.crt COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt AddTrustExternalCARoot.crt > ssl-bundle.crt
2. Store the bundle wherever nginx expects to find it::
mkdir -p /etc/nginx/ssl/example_com/
mv ssl-bundle.crt /etc/nginx/ssl/example_com/
3. Ensure your private key is somewhere nginx can read it, as well.::
mv example_com.key /etc/nginx/ssl/example_com/
4. Make sure your nginx config points to the right cert file and to the private
key you generated earlier::
server {
listen 443;
ssl on;
ssl_certificate /etc/nginx/ssl/example_com/ssl-bundle.crt;
ssl_certificate_key /etc/nginx/ssl/example_com/example_com.key;
# side note: only use TLS since SSLv2 and SSLv3 have had recent vulnerabilities
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
# ...
}
6. Restart nginx.
.. [1] I purchased mine through Namecheap.com.
.. [2] Based on these instructions: http://goo.gl/4zJc8
rst stack_extension.rst
stack_extension.rst
Kafka
https://kafka.apache.org/
https://www.mapr.com/blog/getting-started-sample-programs-apache-kafka-09
http://blog.cloudera.com/blog/2014/09/apache-kafka-for-beginners/
-----------------------------------------------------------------------------------------------------------------------------
Elasticsearch:
https://www.elastic.co/
Mesos:
http://mesos.apache.org/
Spark:
http://spark.apache.org/
Storm:
http://storm.apache.org/
Zookeeper:
https://zookeeper.apache.org/
rst 反应コトハジメ
反应コトハジメ
react.rst
################
React コトハジメ
################
:日時: 2017-01-02
:作: @voluntas
:バージョン: 2.1.0
:URL: https://voluntas.githu.io/
突っ込みは Twitter @voluntas まで。
.. contents::
:depth: 2
概要
====
Erlang/OTP でミドルウェアを作っていると、組み込みの UI が欲しくなる。
ただビジネスの視点からみると開発コストが見合わないことが多く、ついつい後回しになりがちだった。
一念発起して、まずは自分で組み込み UI のプロトタイプを作れる技術を一通り学ぶことにした。
ここでは組み込みの UI を作る際に React を採用した際のノウハウをまとめていきたい。
ミドルウェア開発者も少しはフロントエンドができるようになってもいいんじゃないか？という考えで勉強してみている。
専門家にはなれないだろうけど、フロントエンドエンジニアと会話ができるようになるのが目標。
注意
----
この資料は定期的に更新され、その時の最新版の情報にアップデートされている。
ソースコード
------------
この資料に出てくるコードは以下の URL で取得可能
`voluntas/react-dojo: React コトハジメ <https://github.com/voluntas/react-dojo>`_
変更履歴
========
2017-01-2
---------
- 全体的に新しいライブラリにアップデート
- Step7 として Flow を導入するを追加
対象読者
========
- 大規模商用利用での Erlang/OTP によるプロダクション開発経験あり
- Erlang/OTP コントリビューター
- WebSocket チョットデキル
- JS/CSS/HTML 経験なし
つまり、書いてる本人 (@voluntas) が対象。
選択技術
========
**利用したいと考えている技術**
- React
- Babel
- `Material-UI <http://www.material-ui.com/>`_
- FlowType
- ESLint
- Webpack
- WebSocket
- `Alt <http://alt.js.org/>`_
対象想定
========
ベストプラクティスが、有識者から想定するフレームワークなどでいろいろ変わるというアドバイスを貰ったので事前に定義しておく。
- 通信は WebSocket 99% で非同期通信前提
- サーバ側は Erlang/OTP 前提
- サーバの管理画面として利用される
- リアルタイムにデータが流れてくる
- ログイン/ログアウト以外は非同期で行う
- デザインは `Material-UI <http://www.material-ui.com/#/>`_ ベース
ゴール
======
- React の仕組みを理解する
- Erlang/OTP で書かれたミドルウェアの UI を React + `Material-UI`_ を利用して開発、提供できるようになる
方針
====
- ES6 前提で書く
- 必要になるまで新しい機能は覚えない
- 覚える前にベストプラクティスをできる限り確認する
- 困ったら有識者に教えてもらう
TODO
====
- `Alt - Flux - Managing your state <http://alt.js.org/>`_
- `A Javascript Library For Building Composable And Declarative Charts | React-D3 <http://www.reactd3.org/>`_
- グラフはこれがいいのかなーと思っている
今のところやらない
------------------
- `Grommet コトハジメ <https://gist.github.com/voluntas/38b33f23349e52cdd34ee301c285fae5>`_
- Grommet をざっと調べてみた
- `Reusable Components | React <https://facebook.github.io/react/docs/reusable-components.html>`_
- Flow やれば自動的にやることになりそう
- `reactjs/react-router: A complete routing library for React <https://github.com/reactjs/react-router>`_
- まだ複数画面を管理するところにはいたっていないので、様子見
- `mizchi/flumpt <https://github.com/mizchi/flumpt>`_
- redux よりわかりやすかったのでこれを使う予定
- 子から親の state を更新する際にうまいことできるしくみ、 redux は名前規則でやるが、こちらは親に持たせたメソッドを呼び出すという仕組みらしい
- `UI-Router <https://ui-router.github.io/>`_
準備
====
まず一番最初に困ったのは、そもそも何をしたらいいのかわからなかったこと。とっかかりすら不明。ということで、いろいろ聞いたり調べたりしたので、有識者に相談しつつ自分がこれが良さそうというのを書いていくことにする。
参照
----
基本的には以下のサイトをなぞっているだけ。
- `webpack+React+material-uiの環境を最小手順で作成 - Qiita <http://qiita.com/takaki@github/items/724d97a20d3ae194ded4>`_
- `webpack-dev-serverでReactをES6ですぐに書く - Qiita <http://qiita.com/59naga/items/add00e8df306d7b8d05b>`_
Node.JS と npm を用意する
-------------------------
まずは Node.JS とパッケージマネージャである npm 環境を用意する必要がある。まずはここから。
普段 Erlang しか書いていないので、初 Node.JS 、初 npm 。
MacPorts 派なのでそれでインストール。
::
$ sudo port install nodejs6 npm4
お試し環境を用意する
--------------------
react-dojo というフォルダを作って npm init -y で初期環境を作る
::
$ mkdir react-dojo
$ cd react-dojo
$ mkdir step1
$ cd step1
$ npm init -y
ビルドは今回は webpack を利用する、さらに開発サーバとして webpack-dev-server を利用したいので、インストール。
::
$ npm install --save-dev webpack webpack-dev-server
なにか困ったら::
$ webpack --display-error-details
webpack.config.js
-----------------
webpack.config.js は rebar.config のようなものだろう。ということで、調べて参考にしたのをそのまま活用させて頂く。
query に .babelrc 部分を書けるのがポイント。
.. code-block:: javascript
module.exports = {
devtool: "eval",
entry: {
js: "./src/app.js",
html: "./index.html"
},
output: {
path: __dirname + "/dist",
filename: "./bundle.js"
},
module: {
loaders: [
{
test: /\.html$/,
loader: "file?name=[name].[ext]"
},
{
test: /\.js$/,
loader: "babel-loader",
exclude: /node_modules/,
query: {
presets: ["es2015", "react"]
}
}
]
}
};
app.js というのがベースとなる JS という感じで設定してみた。これで準備ができた。
React
-----
次は今回の目的である React を使うための環境を用意する。
- React
- Babel
npm install --save は実際にパッケージングされる際に利用するもの。 --save-dev は開発時に利用するものとわけるようだ。
::
$ npm install --save react \
react-dom
::
$ npm install --save-dev babel-preset-react \
babel-loader \
babel-core \
babel-preset-es2015 \
file-loader
npm run を設定する
------------------
webpack や webpack-dev-server を使おうとするとこのままではパスが通ってないので利用できない。そんな時は npm init 時に作成した packege.json の scripts をうまく活用するとよいらしい。
以下は 2017-02 の時点で作ったサンプル。
.. code-block:: json
{
"name": "step2",
"version": "2.0.0",
"scripts": {
"build": "webpack --optimize-minimize",
"dev-server": "webpack-dev-server --progress --color"
},
"license": "Apache-2.0",
"devDependencies": {
"babel-core": "^6.21.1",
"babel-loader": "^6.2.10",
"babel-preset-es2015": "^6.18.0",
"babel-preset-react": "^6.16.0",
"file-loader": "^0.9.0",
"webpack": "^1.14.0",
"webpack-dev-server": "^1.16.2"
},
"dependencies": {
"material-ui": "^0.16.6",
"react": "^15.4.1",
"react-dom": "^15.4.1",
"react-tap-event-plugin": "^1.0.0"
}
}
これで npm run build でパッケージング。 npm run dev-server で開発サーバが立ち上がるようになった。おかげでグローバルを汚さなくてすむようになった。
Step1 Hello, world.
===================
フォルダ構成
------------
新しい仕組みを覚えるときに、良く抱える問題の一つとして **フォルダ構成のベストプラクティス** がある。これを一番最初に書いて欲しいと良く思ってしまう。
今回は React.js ということで、そのフォルダ構成を考えてみることにした。まずは最小限に app.js と commponents にその App 用の React コンポーネントをいれる components というフォルダ構成がよさそうということで、以下の構成になった。
::
- index.html
- src/
- app.js
- components
- App.js
index.html を用意する
---------------------
.. code-block:: html
<html>
<head>
<title>React.js Dojo</title>
<meta charset="UTF-8" />
</head>
<body>
<div id="content">Content</div>
<script src="bundle.js" charset="utf-8"></script>
</body>
</html>
React.js では特定のタグに対して、そこにデータを入れていくという仕組みらしいので、 HTML は最小限でいいらしい。
app.js を用意する
-----------------
これがベースとなる JS でここはレンダリングして戻すだけのシンプルな仕組みになる。
.. code-block:: javascript
import React from 'react';
import ReactDOM from 'react-dom';
import App from './components/App.js';
ReactDOM.render(
<App />,
document.getElementById('content')
);
- import
- 'react-dom' の ReactDOM を呼び出すといった、よくある import をしてくれる仕組み
- Erlang では使われない import
- from
- ここはなぜか '' で包む
- ./components/App.js って無理矢理感あってすごい、名前空間とは一体 ...
- ReactDOM.reander
- これがよくわかっていないが、トップレベルで JSX をレンダリングしてくれる何か
- <App />
- components ディレクトリに作った App.js
どうもサンプルの多くがここで Hello, world. をやっているのだが、それだとイメージがつきにくいのであえて components フォルダに App.js ファイルを作ってそちらで Hello, world をしてみることにした。
App.js を用意する
-----------------
`JSX <http://facebook.github.io/jsx/>`_ は今のところは React.Component の render 部分で使われるなんか便利そうな言語という理解することにした。
賛否両論あるようだが、個人的にはまぁ覚えればいいんでしょくらいのスタンス。
.. code-block:: javascript
import React from 'react';
export default class App extends React.Component {
render() {
return (
<h1>Hello, world.</h1>
);
}
}
- export
- 外から呼ぶときに使う、 import/export の仕組み
- default
- 外から呼ぶとき default を設定しておくと自動的にこれが呼ばれるとのこと
- class
- ES6 で使える class の仕組み、便利
- App
- この class の名前
- Extends
- ES6 で使える継承と理解した
- React.Component
- React コンポーネントを作る場合はこれを継承する
- render()
- このコンポーネントの JSX レンダリング部分
ステートレスコンポーネント
^^^^^^^^^^^^^^^^^^^^^^^^^^
この部分だが、 const を使って書き直せるようだ。
render しかなくて、 state を持たないコンポーネントはステートレスコンポーネントと呼ばれるらしい。
.. code-block:: javascript
import React from 'react';
const App = () => {
return <h1>Hello, world.</h1>
};
export {App as default}
export と default の設定は const では使えないため別途設定する必要がある。
webpack-dev-server で動かしてみる
---------------------------------
webpack-dev-server は watchdog と差分ビルドと自前サーバを用意してくれるようだ。これは WebSocket や WebRTC を使う身としては大変ありがたい。
::
$ webpack-dev-server --progress --color
http://localhost:8080/webpack-dev-server/ にアクセスしてみると Hello, world. が表示されています。
まとめ
------
:ソース URL: https://github.com/voluntas/react-dojo/tree/develop/step1
- app.js はほとんど何も書かなくていい
- JSX は HTML っぽいけど、独自定義なので間違えないようにする
- コンポーネントをたくさん作ってうまく render で表示していく仕組み
- Header コンポーネントや、Body コンポーネントを作っていくとイメージしやすいらしい
- webpack の自動ビルドとブラウザリロードはなかなか便利
Step2 はじめての Material-UI
============================
準備
----
Material-UI を利用する場合、 React.js の react-tap-event-plugin をインストールして app.js で呼ぶ必要がある。
この作業は React.js がバージョンが上がれば不要になるようだ。
::
$ npm install --save material-ui \
react-tap-event-plugin
.. code-block:: json
{
"name": "step2",
"version": "2.0.0",
"scripts": {
"build": "webpack --optimize-minimize",
"dev-server": "webpack-dev-server --progress --color"
},
"license": "Apache-2.0",
"devDependencies": {
"babel-core": "^6.21.1",
"babel-loader": "^6.2.10",
"babel-preset-es2015": "^6.18.0",
"babel-preset-react": "^6.16.0",
"file-loader": "^0.9.0",
"webpack": "^1.14.0",
"webpack-dev-server": "^1.16.2"
},
"dependencies": {
"material-ui": "^0.16.6",
"react": "^15.4.1",
"react-dom": "^15.4.1",
"react-tap-event-plugin": "^2.0.1"
}
}
ボタンを設定してみる
--------------------
`Material-UI`_ はボタンだけを置くことを目標にします。まずはボタンを置いてみることから。
App.js を書き換えていく。必要になりそうな Material-UI のパーツを render() に書いていけば良い。
まずボタンを配置するだけ。
.. code-block:: javascript
import React from 'react';
import ReactDOM from 'react-dom';
// この二行を追加
import injectTapEventPlugin from 'react-tap-event-plugin';
injectTapEventPlugin();
import MuiThemeProvider from 'material-ui/styles/MuiThemeProvider';
import App from './components/App.js';
ReactDOM.render(
<MuiThemeProvider>
<App />
</MuiThemeProvider>,
document.getElementById('content')
);
あとは MuiThemeProvider で <App /> を挟む。まずはこれで Material-UI を使う準備はできた。
MuiThemeProvider はテーマらしく、好きに変えられる仕組みらしい。
.. code-block:: javascript
import React from 'react';
import RaisedButton from 'material-ui/RaisedButton';
export default class App extends React.Component {
render() {
return (
<div>
<h1>Hello, world.</h1>
<RaisedButton label="Hello, world." />
</div>
);
}
}
まとめ
------
:ソース URL: https://github.com/voluntas/react-dojo/tree/develop/step2
Step3 Material-UI のコンポーネントを連携させてみる
==================================================
いくつかコンポーネントを追加してみる
TextField と Snackbar だ。
- http://www.material-ui.com/#/components/text-field
- http://www.material-ui.com/#/components/snackbar
TextField の説明は省略。Snackbar は簡易的なポップアップで時間が立ったら消える。
まずは完全な App.js のコードをここに貼っておく。
.. code-block:: javascript
import React from 'react';
import TextField from 'material-ui/TextField';
import RaisedButton from 'material-ui/RaisedButton';
import Snackbar from 'material-ui/Snackbar';
export default class App extends React.Component {
constructor(props) {
super(props);
this.state = {
open: false,
value: '',
};
}
handleTouchTap() {
this.setState({
open: true,
});
}
handleRequestClose() {
this.setState({
open: false,
});
}
handleChange(e) {
this.setState({
value: e.target.value,
});
}
render() {
return (
<div>
<h1>Hello, world.</h1>
<TextField
hintText="message"
value={this.state.value}
onChange={this.handleChange.bind(this)}
/>
<br/>
<RaisedButton
onTouchTap={this.handleTouchTap.bind(this)}
label='Hello, world'
/>
<Snackbar
open={this.state.open}
message={this.state.value}
autoHideDuration={1000}
onRequestClose={this.handleRequestClose.bind(this)}
/>
</div>
);
}
}
constructor
-----------
このコンポーネントの初期化した時に呼ばれる値。props は親コンポーネントから送られてくる値が入っているのでそのまま使うための super 。
this.state はこのコンポーネント自身が持つ状態。gen_server の #state{} みたいなもの。 init/1 で #state{} 定義するのと同じ。
.. code-block:: javascript
import TextField from 'material-ui/TextField';
export default class App extends React.Component {
constructor(props) {
super(props);
this.state = {
open: false,
value: '',
};
}
...
}
ここでは Snackbar を出現させるかどうかの open: boolean() と TextField に入力した値を持っておく value: string() を用意する。それぞれ初期値も与えておく。
TextField
---------
TextField は実際の入力した値の部分と、その値が変化した部分がコンポーネントと連携している
.. code-block:: javascript
<TextField
hintText="message"
value={this.state.value}
onChange={this.handleChange.bind(this)}
/>
- hintText は TextField に後ろにうっすらと何を入力すればいいか、書かれているメッセージを指定できる
- value は constructor で定義した値をコンポーネントから状態を引っ張ってきている
- onChange はコンポーネントに定義した handleChange を呼び出している
- このテキストが変化したらこの関数を呼び出せという内容
.. code-block:: javascript
handleChange(e) {
this.setState({
value: e.target.value,
});
}
handleChange を見ていくと event.target.value という感じで先程の TextField に値が入力されるたびに setState されている。 this.handleChange だが、 bind(this) は JSX の世界とコンポーネントの世界を連動させているものという認識でまずよさそうだ。
Snackbar
--------
Snackbar は起動する部分と、閉じるタイミングの部分、そしてメッセージの部分がコンポーネントと連動している。
.. code-block:: javascript
<Snackbar
open={this.state.open}
message={this.state.value}
autoHideDuration={1000}
onRequestClose={this.handleRequestClose.bind(this)}
/>
- open はこの Snackbar を開く部分
- this.state.open で初期値は false になっている、ここが true になると Snackbar が起動する
- message は Snackbar に表示される文字列
- autoHideDuration は自動で消える時間、ここでは 1000 を指定してる
- onRequestClose は閉じるタイミングで実行する処理を指定できる
.. code-block:: javascript
handleRequestClose() {
this.setState({
open: false,
});
}
handleRequestClose では setState で open を false にしている、ここを true にするとまた上がってきてしまう。ここでも this.handleRequestClose.bind(this) だが、用意されたメソッド以外はこうやって bind していく必要がある。
RaisedButton
------------
RaisedButton はそのボタンがクリックされたタイミングがコンポーネントと連動している
.. code-block:: javascript
<RaisedButton
onTouchTap={this.handleTouchTap.bind(this)}
label='Hello, world'
/>
- onTouchTap はそのボタンが押されたら handleTouchTap を実行する
.. code-block:: javascript
handleTouchTap() {
this.setState({
open: true,
});
}
setState で open を true にすることで Snackbar を起動している
まとめ
------
:ソース URL: https://github.com/voluntas/react-dojo/tree/develop/step3
- Material-UI のコンポーネントを追加するのは難しくない
- Material-UI コンポーネント同士の連携も、App.js の State に集約されているため、そこが共通の値として動作しているので扱いやすい
- App.js のメソッドと Material-UI のコンポーネントを連携させる場合は .bind(this) を利用すればよい
Step4 WebSocket を導入する
==========================
コンポーネント同士の連携が確認できたところで、これをサーバサイドとのやり取りに置き換えていく。
具体的にはボタンを押したら WebSocket で TextField の値がサーバに送られ、サーバは echo でそのまま値を返してきて、その値を Snackbar に表示させるというものだ。
結果はコンポーネントの連携と変わらないが、通信が含まれるので、書き方が変わってくる。
今回 WebSocket サーバを用意するのを省略するため wss://echo.websocket.org を利用させてもらうことにする。
参考
----
- `React.jsのComponent Lifecycle - Qiita <http://qiita.com/koba04/items/66e9c5be8f2e31f28461>`_
Component Lifecycle
-------------------
React には Component Lifecycle というものがあるようで、よく使うのが componentDidMount と componentWillUnmount のようだ。
これらは起動時にと終了時に一回だけ呼ばれるもので、 WebSocket の場合は componentDidMount で new して componentWillUnmount で close する感じになるのだと思う。
componentDidMount
-----------------
.. code-block:: javascript
constructor(props) {
super(props);
this.state = {
// ws を定義する
ws: null,
// 送られてきたメッセージ格納用
message: '',
open: false,
value: '',
};
}
componentDidMount() {
var ws = new WebSocket("wss://echo.websocket.org");
ws.onmessage = this.handleMessage.bind(this);
this.setState({ws: ws});
}
componentDidMount には WebSocket を new し、その後 onmessage メソッドを WebSocket のコールバックに登録している。
.. code-block:: javascript
handleMessage(msg) {
this.setState({
// 送られてきたメッセージを格納し
message: msg.data,
// 状態を open へ
open: true,
});
}
componentWillUnmount
--------------------
componentWillUnmount はこのコンポーネントが DOM から削除されるタイミングに呼ばれるので、丁寧に WebSocket を close しておく。
.. code-block:: javascript
componentWillUnmount() {
this.state.ws.close();
}
RaisedButton
------------
ボタンを押したら、 WebSocket 経由でメッセージをサーバへ送る。
.. code-block:: javascript
handleTouchTap() {
this.state.ws.send(this.state.value);
}
handleTouchTap が実行されたら state にいる WebSocket から state にある TextField に入力した値を送るというのを handleTouchTap に書く。
Snackbar
--------
Snackbar の変更点は一つだけで、サーバから送られてきたメッセージを表示するようにする。
.. code-block:: javascript
<Snackbar
open={this.state.open}
message={this.state.message}
autoHideDuration={1000}
onRequestClose={this.handleRequestClose.bind(this)}
/>
message を this.state.message に変更するだけだ。これでサーバから送られてきたメッセージを読み込み、表示するようになる。
まとめ
------
:ソース URL: https://github.com/voluntas/react-dojo/tree/develop/step4
まずは全体のコードを。 App.js のみ変更している。
動作の見た目は全然変わらないが、WebSocket サーバ送ったメッセージを非同期で受け取り、そのメッセージが Snackbar に表示されるという仕組み。
.. code-block:: javascript
import React from 'react';
import TextField from 'material-ui/TextField';
import RaisedButton from 'material-ui/RaisedButton';
import Snackbar from 'material-ui/Snackbar';
export default class App extends React.Component {
constructor(props) {
super(props);
this.state = {
ws: null,
message: '',
open: false,
value: '',
};
}
componentDidMount() {
var ws = new WebSocket("wss://echo.websocket.org");
ws.onmessage = this.handleMessage.bind(this);
this.setState({ws: ws});
}
componentWillUnmount() {
this.state.ws.close();
}
handleMessage(msg) {
this.setState({
message: msg.data,
open: true,
});
}
handleChange(e) {
this.setState({
value: e.target.value,
});
}
handleTouchTap() {
this.state.ws.send(this.state.value);
}
handleRequestClose() {
this.setState({
open: false,
});
}
render() {
return (
<div>
<h1>Hello, world.</h1>
<TextField
hintText="message"
value={this.state.value}
onChange={this.handleChange.bind(this)}
/>
<br/>
<RaisedButton
onTouchTap={this.handleTouchTap.bind(this)}
label='Hello, world'
/>
<Snackbar
open={this.state.open}
message={this.state.message}
autoHideDuration={1000}
onRequestClose={this.handleRequestClose.bind(this)}
/>
</div>
);
}
}
- this.state は便利で、そこにおいておけばあとはその変更によって各コンポーネントが動作をしてくれるのはとても良い
- コールバックというのをあまり意識しないのはおそらく state の値の変更での発火だからかもしれない
Step5 ESLint を導入する
=======================
コーディング規約は便利なので、それに簡単に気付くための ESLint を導入することにした。
.eslintrc.js
--------------
ESLint 関連で今回 npm install したのは以下の通り
::
$ npm install --save-dev eslint \
babel-eslint \
eslint-loader \
eslint \
eslint-plugin-babel \
eslint-plugin-material-ui \
eslint-plugin-react
.eslintrc は eslint --init で質問に答えて設定してみた。途中で形式を聞かれたので、 js にすればコメントかけてよさそうという判断で js にしてみた。
パスが通ってないので直接実行する::
$ node_modules/eslint/bin/eslint.js --init
? How would you like to configure ESLint? Answer questions about your style
? Are you using ECMAScript 6 features? Yes
? Are you using ES6 modules? Yes
? Where will your code run? Browser
? Do you use CommonJS? Yes
? Do you use JSX? Yes
? Do you use React Yes
? What style of indentation do you use? Spaces
? What quotes do you use for strings? Single
? What line endings do you use? Unix
? Do you require semicolons? Yes
? What format do you want your config file to be in? JavaScript
Successfully created .eslintrc.js file in react-dojo/step5
生成された .eslintrc.js ファイル
.. code-block:: javascript
module.exports = {
"env": {
"browser": true,
"commonjs": true,
"es6": true
},
"extends": "eslint:recommended",
"parserOptions": {
"ecmaFeatures": {
"experimentalObjectRestSpread": true,
"jsx": true
},
"sourceType": "module"
},
"plugins": [
"react"
],
"rules": {
"indent": [
"error",
4
],
"linebreak-style": [
"error",
"unix"
],
"quotes": [
"error",
"single"
],
"semi": [
"error",
"always"
]
}
};
webpack.config.js
-----------------
webpack から ESLint が使えるようにするには module と eslint という項目を増やしておく。 preLoaders なのは、 ESLint がお勧めしていたのでそのまま使った。
webpack.config.js で追加したのは以下の通り。
.. code-block:: javascript
module.exports = {
devtool: "eval",
entry: {
js: "./src/app.js",
html: "./index.html"
},
output: {
path: __dirname + "/dist",
filename: "./bundle.js"
},
module: {
// ここを追加
preLoaders: [
{test: /\.js$/, loader: "eslint-loader", exclude: /node_modules/}
],
loaders: [
{
// file-loader を利用している
test: /\.html$/,
loader: "file?name=[name].[ext]"
},
{
test: /\.js$/,
loader: "babel-loader",
exclude: /node_modules/,
query: {
presets: ["es2015", "react"]
}
}
]
},
// ここを追加
eslint: {
configFile: '.eslintrc.js'
}
};
package.json
------------
.. code-block:: javascript
{
"name": "step5",
"version": "2.0.0",
"scripts": {
"lint": "eslint src",
"lint:fix": "eslint src --fix",
"build": "webpack --optimize-minimize",
"dev-server": "webpack-dev-server --progress --color"
},
"license": "Apache-2.0",
"devDependencies": {
"babel-core": "^6.21.0",
"babel-eslint": "^7.1.1",
"babel-loader": "^6.2.10",
"babel-preset-es2015": "^6.18.0",
"babel-preset-react": "^6.16.0",
"eslint": "^3.12.2",
"eslint-loader": "^1.6.1",
"eslint-plugin-babel": "^3.3.0",
"eslint-plugin-material-ui": "^1.0.1",
"eslint-plugin-react": "^6.8.0",
"file-loader": "^0.9.0",
"webpack": "^1.14.0",
"webpack-dev-server": "^1.16.2"
},
"dependencies": {
"material-ui": "^0.16.6",
"react": "^15.4.1",
"react-dom": "^15.4.1",
"react-tap-event-plugin": "^1.0.0"
}
}
.eslintrc.js を設定する
-----------------------
jsx の部分は `Material-ui の .eslintrc.js <https://github.com/callemall/material-ui/blob/master/.eslintrc.js>`_ を参考にしてみた。
.. code-block:: javascript
module.exports = {
"env": {
"browser": true,
"es6": true
},
"extends": "eslint:recommended",
"parserOptions": {
"ecmaFeatures": {
"experimentalObjectRestSpread": true,
"jsx": true
},
"sourceType": "module"
},
"plugins": [
"babel",
"react",
"material-ui",
],
"rules": {
'react/jsx-boolean-value': [
'error',
'always'
],
'react/jsx-closing-bracket-location': 'error',
'react/jsx-curly-spacing': 'error',
'react/jsx-equals-spacing': 'error',
'react/jsx-filename-extension': [
'error', {
extensions: ['.js']
}
],
'react/jsx-first-prop-new-line': [
'error',
'multiline'
],
'react/jsx-handler-names': 'error',
'react/jsx-no-comment-textnodes': 'error',
'react/jsx-no-duplicate-props': 'error',
'react/jsx-no-undef': 'error',
'react/jsx-pascal-case': 'error',
'react/jsx-uses-react': 'error',
'react/jsx-uses-vars': 'error',
'react/jsx-wrap-multilines': 'error',
"indent": [
"error",
4
],
"linebreak-style": [
"error",
"unix"
],
"quotes": [
"error",
"single"
],
"semi": [
"error",
"always"
]
}
};
適用する
--------
eslint --fix を利用すればある程度までは自動で修正してくれるらしい。package.json の scripts に仕込んでおいた。
::
$ npm run lint:fix
`WebSocket を導入する`_ のコードに対して修正してみた。
まとめ
------
:ソース URL: https://github.com/voluntas/react-dojo/tree/develop/step5
- とても細かく設定可能なので、大変そうだがコピペで済みそう
- 導入までの距離がとても短い
- webpack-dev-server ともさくっと連絡して良い
- lint を採用するときのスタンダードがわかりにくいので、どうするべきか
- `eslint-config-airbnb <https://www.npmjs.com/package/eslint-config-airbnb>`_
- この辺をお手本にするといいのかもしれない
Step6 WebSocket から送られてくるデータを表示する
===================================================
目的
----
今回は Erlang/OTP で書いた WebSocket サーバを用意した。接続すると 1 秒毎に値を返してくる。これを画面にだらだらと表示していく仕組みを実現していきたい。
ちなみに、今回は Material-UI は利用していない。
準備
----
Erlang/OTP で、 WebSocket サーバを用意しておいた。
https://github.com/voluntas/react-dojo/tree/develop/step6/step6-server
Erlang/OTP 19.1 がインストールされていれば make dev で起動するようになっている。
このサーバは接続すると 1 秒毎に値が送られてくる。送られてくるのは以下のような値だ。
.. code-block:: json
{
"count": 1,
"uuid": "dfb79b00-fff9-42fc-9f72-6dfe1ae26a5f"
}
count は毎秒毎に +1 され、 UUID は毎回変わる。この count は接続毎にカウントされていく。
実装
----
.. code-block:: javascript
import React from "react";
import ReactDOM from "react-dom";
import App from "./components/App";
ReactDOM.render(
<App />,
document.getElementById("content")
);
.. code-block:: javascript
import React from "react";
import ReactDOM from "react-dom";
export default class App extends React.Component {
constructor(props) {
super(props);
this.state = {
ws: null,
messages: [],
};
}
componentDidMount() {
let ws = new WebSocket("ws://127.0.0.1:8000/ws");
ws.onmessage = this.handleMessage.bind(this);
this.setState({ws: ws});
}
componentWillUnmount() {
this.state.ws.close();
}
componentDidUpdate() {
let node = ReactDOM.findDOMNode(this.refs.messages);
// XXX: これは .. ありなのだろうか
window.scrollTo(0, node.scrollHeight);
}
handleMessage(event) {
let message = JSON.parse(event.data);
this.state.messages.push(message);
this.setState({
messages: this.state.messages
});
}
render() {
let messages = this.state.messages.map(v => {
return <div key={v.uuid} >{v.uuid}: {v.count}</div>;
});
return (
<div ref="messages">{messages}</div>
);
}
}
- constructor で state を ws と messages を定義
- messages は送られてきた JSON を格納する
- componentDidMount で WebSocket を接続している
- ws://127.0.0.1:8000/ws は Erlang/OTP で設定した値
- setState で ws を保持
- componentDidUpdate でコンポーネントが更新された時にスクロールを一番したにする
- ReactDOM.findDOMNode で messages の DOM を取得する
- this.refs で render (JSX) 部分で名前を付けて引っ張れる
- <div ref="messages"> は this.refs.messages で引っ張れる
- ちなみに window.scrollTo でいいのかわからない
- handleMessage は onmessage のコールバックで、 WebSocket からメッセージが送られてきたら呼ばれる
- 送られてくる JSON をパースして、そのまま this.state.messages に push で突っ込む
- render は JSX で表示する部分
- messages を map で <div /> にする
- key を指定することで差分更新になる
- return で ref を指定しているのは this.refs で引っ張れるようにしているため
これで、あとは webpack-dev-server を起動すれば、画面に WebSocket サーバから送られてくる値が淡々と表示され、続ける。スクロールは一番下に固定され続ける。
おまけ
------
console.debug を使おうと思ったら eslint でエラーになった。
eslint の設定に以下を追加する必要があるらしい。
.. code-block:: js
'no-console': [
"error",
{
allow: [
"debug"
]
}
],
まとめ
------
- JSX に慣れていない、 JSX の癖をもっとつかまないと上手く利用するのは難しそう
- JS の DOM を意識することはあまりなかったがはやり少しでも動きが入ると必要になる
- そろそろこちらからリクエストを送って何かハンドリングをさせてみるという処理が必要になりそう
Flow を導入してみる
===================
`Flow <https://flowtype.org/>`_ は静的型解析ツールで OCaml で書かれている。既存のアプリに導入しやすい後出しできる型解析ツール。
- `Flow を導入してみる - botchy note <http://negi-works.hatenablog.com/entry/2016/12/01/230621>`_
- `実践投入してわかったflowtypeのメリデメ // Speaker Deck <https://speakerdeck.com/shinout/shi-jian-tou-ru-sitewakatutaflowtypefalsemerideme>`_
`/* @flow */` を入れるだけで始められるのはとても良いらしい、ということで試してみることにする。
flow をセットアップする
-----------------------
package.json は step5 からの流用、そこに二つほどライブラリを追加。
::
$ npm install --save-dev flow-bin \
babel-plugin-transform-flow-strip-types
scripts 部分に flow コマンドを追加。
package.json::
{
"name": "step5",
"version": "2.0.0",
"scripts": {
"flow": "flow",
"lint": "eslint src",
"lint:fix": "eslint src --fix",
"build": "webpack --optimize-minimize",
"dev-server": "webpack-dev-server --progress --color"
},
"license": "Apache-2.0",
"devDependencies": {
"babel-core": "^6.21.0",
"babel-eslint": "^7.1.1",
"babel-loader": "^6.2.10",
"babel-plugin-transform-flow-strip-types": "^6.21.0",
"babel-preset-es2015": "^6.18.0",
"babel-preset-react": "^6.16.0",
"eslint": "^3.12.2",
"eslint-loader": "^1.6.1",
"eslint-plugin-babel": "^3.3.0",
"eslint-plugin-material-ui": "^1.0.1",
"eslint-plugin-react": "^6.8.0",
"file-loader": "^0.9.0",
"flow-bin": "^0.37.4",
"webpack": "^1.14.0",
"webpack-dev-server": "^1.16.2"
},
"dependencies": {
"material-ui": "^0.16.6",
"react": "^15.4.1",
"react-dom": "^15.4.1",
"react-tap-event-plugin": "^1.0.0"
}
}
flow init
---------
.flowconfig を生成する::
$ node_modules/flow-bin/flow-osx-v0.37.4/flow init
.flowconfig の ignore は最低限でやるべきっぽい。 node_modules を全部対象にすると、そんなライブラリみつからんって怒られる
.flowconfig::
[ignore]
.*node_modules/fbjs.*
[include]
[libs]
[options]
flow + webpack
--------------
このタイミングではよくわからなかったので、省略する。
とりあえず npm run flow コマンドを叩いて利用するで終わらせることにする。
flow の loader の上手い使い方がわかったらそれを後で書く
babel + flow + webpack
----------------------
babel-plugin-transform-flow-strip-types これをセットアップして webpack に設定を入れる必要があるようだ。
おそらく flow 型設定が書いてあってもエラーにならないようにするためのプラグインだろう。
plugins を追加して transform-flow-strip-types を追加することにした。
::
module: {
preLoaders: [
{test: /\.js$/, loader: "eslint-loader", exclude: /node_modules/}
],
loaders: [
{
// file-loader を利用している
test: /\.html$/,
loader: "file?name=[name].[ext]"
},
{
test: /\.js$/,
loader: "babel-loader",
exclude: /node_modules/,
query: {
presets: ["es2015", "react"],
plugins: ["transform-flow-strip-types"]
}
}
]
},
babel + flow + react
---------------------
`Flow | React <https://flowtype.org/docs/react.html>`_
公式に書いてあるので、まずは src/app.js と src/components/App.js に /* @flow */ を付けてみることにする。
実行してみたら、凄い勢いでエラーになった。
**書いている途中です ...**
Step1-1 ライブラリのアップデートをチェックする
==============================================
**ここは毎回変わってしまうので参考程度に見てもらえれば**
ライブラリのアップデートの追従はとても大切なことだ、ただやり方がわからないので調べることにした。
package.json をアップデートするには npm update するというのはわかったのだが、最新版に追従したいとき一つ一つチェックしていくのは面倒くさい。 npm update を実行すれば良いということがわかった。
この際、 npm update -D と npm update -S という順番にやる必要がある。 -D は --save-dev の略で、 -S は --save の略だ。
-S を先にしてしまうと devDependencies でも dependencies に入ってしまうとのことを Twitter で教えて頂いた。ありがたい。
::
$ npm update -D
- babel-plugin-syntax-async-functions@6.13.0 node_modules/babel-plugin-syntax-async-functions
- source-map@0.1.32 node_modules/source-map-support/node_modules/source-map
step1-1@1.0.0 /react-dojo/step1-1
├── babel-core@6.17.0
├── babel-preset-es2015@6.16.0
├── babel-preset-react@6.16.0
└── webpack-dev-server@1.16.2
::
% npm update -S
step1-1@1.0.0 /react-dojo/step1-1
├── react@15.3.2
└── react-dom@15.3.2
npm WARN step1-1@1.0.0 No description
npm WARN step1-1@1.0.0 No repository field.
無事 package.json がアップデートされている。
::
diff --git a/step1-1/package.json b/step1-1/package.json
index 70b371f..a1734fd 100644
--- a/step1-1/package.json
+++ b/step1-1/package.json
@@ -7,16 +7,16 @@
},
"license": "Apache-2.0",
"devDependencies": {
- "babel-core": "^6.14.0",
+ "babel-core": "^6.17.0",
"babel-loader": "^6.2.5",
- "babel-preset-es2015": "^6.14.0",
- "babel-preset-react": "^6.11.1",
+ "babel-preset-es2015": "^6.16.0",
+ "babel-preset-react": "^6.16.0",
"file-loader": "^0.9.0",
"webpack": "^1.13.2",
- "webpack-dev-server": "^1.15.2"
+ "webpack-dev-server": "^1.16.2"
},
"dependencies": {
- "react": "^15.3.1",
- "react-dom": "^15.3.1"
+ "react": "^15.3.2",
+ "react-dom": "^15.3.2"
}
}
ということで、無事期待する結果を得る方法を学ぶことができた。
まとめ
------
:ソース URL: https://github.com/voluntas/react-dojo/tree/develop/step1-1
標準でちゃんとチェックツールが入っているのでとても便利。管理が膨大になるとどこかのバージョンのライブラリがおかしくなってしんどくなりそうな感じもあるが、それはそのとき考える。
メジャーバージョンアップに関しては自前でやる必要がある
おまけ
------
`Greenkeeper <https://greenkeeper.io/>`_
ローカルでやるのが面倒だなと感じる人は、自動でアップデートを見て Pull-Request を送ってくれるサービスとのことを Twitter にて教えて頂いた。
これは大変賢いサービスだし、とても良いように思える。もし必要になったら導入していきたい。
rst 设置Django，nginx和uwsgi以及python
设置Django，nginx和uwsgi以及python
gistfile1.rst
This document has now been incorporated into the uWSGI documentation::
http://uwsgi-docs.readthedocs.org/en/latest/tutorials/Django_and_nginx.html
******************************
Set up Django, nginx and uwsgi
******************************
Steps with explanations to set up a server using:
* virtualenv
* Django
* nginx
* uwsgi
Concept
=======
nginx will face the outside world. It will serve media files (images, CSS, etc) directly from the file system. However, it can't talk directly to Django applications; it needs something that will run the application, feed it requests from the web, and return responses.
That's uwsgi's job. uwsgi will create a Unix socket, and serve responses to nginx via the uwsgi protocol - the socket passes data in both directions::
the outside world <-> nginx <-> the socket <-> uwsgi
Before you start
================
virtualenv
----------
Make sure you are in a virtualenv - you will install a system-wide uwsgi later.
Django
------
I am assuming Django 1.4. It automatically creates a wsgi module when you create a project. If you're using an earlier version, you will have to find a Django wsgi module for your project.
Note that I'm also assuming a Django 1.4 project structure, in which you see paths like::
/path/to/your/project/project/
(i.e. it creates nested directories with the name of your project). Adjust the examples if you using an earlier Django.
It will also be helpful if you are in your Django project's directory. If you don't have one ready, just create a directory for now.
About the domain and port
-------------------------
I'll call your domain example.com. Substitute your own FQDN or IP address.
Throughout, I'm using port 8000. You can use whatever port you want of course, but I have chosen this one so it doesn't conflict with anything a web server might be doing already.
Basic uwsgi intallation and configuration
=========================================
Install uwsgi
-------------
::
pip install uwsgi
Basic test
----------
Create a file called test.py::
# test.py
def application(env, start_response):
start_response('200 OK', [('Content-Type','text/html')])
return "Hello World"
Run::
uwsgi --http :8000 --wsgi-file test.py
The options mean:
http :8000
use protocol http, port 8000
wsgi-file test.py
load the specified file
This should serve a hello world message directly to the browser on port 8000. Visit::
http://example.com:8000
to check.
Test your Django project
------------------------
Now we want uwsgi to do the same thing, but to run a Django site instead of the test.py module.
But first, make sure that your project actually works! Now you need to be in your Django project directory.
::
python manage.py runserver 0.0.0.0:8000
Now run it using uwsgi::
uwsgi --http :8000 --chdir /path/to/your/project --module project.wsgi --virtualenv /path/to/virtualenv
The options mean:
chdir /path/to/your/project
use your Django project directory as a base
module project.wsgi
i.e. the Python wsgi module in your project
virtualenv /path/to/virtualenv
the virtualenv
There is an alternative to using the `--module` option, by referring instead to the wsgi *file*::
wsgi-file /path/to/your/project/project/wsgi.py
i.e. the system file path to the wsgi.py file
Point your browser at the server; if the site appears, it means uwsgi can serve your Django application from your virtualenv. Media/static files may not be served properly, but don't worry about that.
Now normally we won't have the browser speaking directly to uwsgi: nginx will be the go-between.
Basic nginx
===========
Install nginx
-------------
The version of Nginx from Debian stable is rather old. We'll install from backports.
::
sudo pico /etc/apt/sources.list # edit the sources list
Add::
# backports
deb http://backports.debian.org/debian-backports squeeze-backports main
Run::
sudo apt-get -t squeeze-backports install nginx # install nginx
sudo /etc/init.d/nginx start # start nginx
And now check that the server is serving by visiting it in a web browser on port 80 - you should get a message from nginx: "Welcome to nginx!"
Configure nginx for your site
-----------------------------
Check that your nginx has installed a file at `/etc/nginx/uwsgi_params`. If not, copy http://projects.unbit.it/uwsgi/browser/nginx/uwsgi_params to your directory, because nginx will need it. Easiest way to get it::
wget http://projects.unbit.it/uwsgi/export/3fab63fcad3c77e7a2a1cd39ffe0e50336647fd8/nginx/uwsgi_params
Create a file called nginx.conf, and put this in it::
# nginx.conf
upstream django {
# connect to this socket
# server unix:///tmp/uwsgi.sock; # for a file socket
server 127.0.0.1:8001; # for a web port socket
}
server {
# the port your site will be served on
listen 8000;
# the domain name it will serve for
server_name .example.com; # substitute your machine's IP address or FQDN
charset utf-8;
#Max upload size
client_max_body_size 75M; # adjust to taste
# Django media
location /media {
alias /path/to/your/project/project/media; # your Django project's media files
}
location /static {
alias /path/to/your/project/project/static; # your Django project's static files
}
# Finally, send all non-media requests to the Django server.
location / {
uwsgi_pass django;
include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
}
}
Symlink to this file from /etc/nginx/sites-enabled so nginx can see it::
sudo ln -s ~/path/to/your/project/nginx.conf /etc/nginx/sites-enabled/
Basic nginx test
----------------
Restart nginx::
sudo /etc/init.d/nginx restart
Check that media files are being served correctly:
Add an image called media.png to the /path/to/your/project/project/media directory
Visit
http://example.com:8000/media/media.png
If this works, you'll know at least that nginx is serving files correctly.
nginx and uwsgi and test.py
===========================
Let's get nginx to speak to the hello world test.py application.
::
uwsgi --socket :8001 --wsgi-file test.py
This is nearly the same as before, except now we are not using http between uwsgi and nginx, but the (much more efficient) uwsgi protocol, and we're doing it on port 8001. nginx meanwhile will pass what it finds on that port to port 8000. Visit:
http://example.com:8000/
to check.
Meanwhile, you can try to have a look at the uswgi output at:
http://example.com:8001/
but quite probably, it won't work because your browser speaks http, not uwsgi.
Using sockets instead of ports
==============================
It's better to use Unix sockets than ports - there's less overhead.
Edit nginx.conf.
uncomment
server unix:///tmp/uwsgi.sock;
comment out
server 127.0.0.1:8001;
and restart nginx.
Runs uwsgi again::
uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py
Try http://example.com:8000/ in the browser.
If that doesn't work
--------------------
Check your nginx error log(/var/log/nginx/error.log). If you see something like::
connect() to unix:///path/to/your/project/uwsgi.sock failed (13: Permission denied)
then probably you need to manage the permissions on the socket (especially if you are using a file not in /tmp as suggested).
Try::
uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py --chmod-socket=644 # 666 permissions (very permissive)
or::
uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py --chmod-socket=664 # 664 permissions (more sensible)
You may also have to add your user to nginx's group (probably www-data), or vice-versa, so that nginx can read and write to your socket properly.
Running the Django application with uswgi and nginx
===================================================
Let's run our Django application::
uwsgi --socket /tmp/uwsgi.sock --chdir /path/to/your/project --module project.wsgi --virtualenv /path/to/virtualenv --chmod-socket=664
Now uwsgi and nginx should be serving up your Django application.
a uwsgi .ini file for our Django application
============================================
Deactivate your virtualenv::
deactivate
and install uwsgi system-wide::
sudo pip install uwsgi
We can put the same options that we used with uwsgi into a file, and then ask uwsgi to run with that file::
# django.ini file
[uwsgi]
# master
master = true
# maximum number of processes
processes = 10
# the socket (use the full path to be safe)
socket = /tmp/uwsgi.sock
# with appropriate permissions - *may* be needed
# chmod-socket = 664
# the base directory
chdir = /path/to/your/project
# Django's wsgi file
module = project.wsgi
# the virtualenv
home = /path/to/virtualenv
# clear environment on exit
vacuum = true
And run uswgi using the file::
uwsgi --ini django.ini
Note:
--ini django.ini
use the specified .ini file
Test emperor mode
=================
uwsgi can run in 'emperor' mode. In this mode it keeps an eye on a directory of uwsgi config files, and spawns instances ('vassals') for each one it finds.
Whenever a config file is amended, the emperor will automatically restart the vassal.
::
# symlink from the default config directory to your config file
sudo ln -s /path/to/your/project/django.ini /etc/uwsgi/vassals/
# run the emperor as root
sudo uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master
The options mean:
emperor /etc/uwsgi/vassals
look there for vassals (config files)
uid www-data
run as www-data once we've started
gid www-data
run as www-data once we've started
Check the site; it should be running.
Make uwsgi startup when the system boots
========================================
The last step is to make it all happen automatically at system startup time.
Edit /etc/rc.local and add::
/usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master
before the line "exit 0".
And that should be it!