Mathew Brady Accused of Insider Trading While Visiting Grant in Virginia

By Bob Zeller · Originally published March 2023 · pp. 20–22

Mathew Brady Accused of Insider Trading While Visiting Grant in Virginia

by Bob Zeller This story was originally published in the April 2014 issue of Battlefield Photographer, the journal of The Center for Civil War Photography

On Oct. 31, 1864, Assistant Secretary of War Charles A. Dana scrawled his signature, “C.A. Dana” on the outside of a highly sensitive three-page letter that would soon be making its way to headquarters of the Union Army at City Point, Va. Above Dana’s signature appeared these words: “Confidentially referred to Lt. Gen. U.S. Grant.” Most pieces of correspondence arriving at army headquarters in City Point, Va., were routinely recorded into the army’s official register of correspondence and alphabetically cross-indexed. But this letter was not a routine piece of correspondence. It survives to this day at the National Archives, lumped among letters that came to army headquarters but were never registered.1 The letter was written by LaFayette C. Baker, the war department’s spy master and Provost-Marshal of the War Department, who would gain the greatest fame of his star-crossed career in the pursuit and capture of John Wilkes Booth after the assassination of Abraham Lincoln. Baker, one of the most controversial figures in the North during the war, had a healthy appetite for rooting out scandal, real or otherwise, and seemed to see a conspiracy lurking in every dark corner.

This rarely seen sketch by Civil War artist Charles Wellington Reed shows a photographer preparing to try to take Gen. Ulysses S. Grant’s photo at his headquarters in City Point, Va., as Grant’s wife helps him into his jacket. Reed’s hand-written caption reads, “’Now, ‘Ulis’ let him do it, it’s the way he makes his living, it won’t take but a minute,’ Mrs. U.S. Grant said.” (Library of Congress)

photo market. But many of the negatives of these images have survived and are preserved at the National Archives or the Library of Congress.3 If Baker’s report is to be believed, Brady had an ulterior motive in visiting Grant that went far beyond photography, and was unusually determined to make that visit happen. Baker alleged that when Brady requested permission to visit the army in 1864, it was “refused at the War Department.” After this denial, “Brady then applied to the President, who also refused to grant his request,” Baker wrote. Undeterred, Brady wrote to Grant’s wife, Julia, “to use her influence with the General to procure the desired pass,” he

Assistant Secretary of War Charles A. Dana was photographed by Brady in Virginia in June 1864, the same time the alleged insider information scheme was supposedly taking place. (Library of Congress)

“If Baker’s allegations

Now his sights were set squarely on the most heralded photographer in the North, Mathew B. Brady, the official photographer for U.S. Grant and the headquarters command of the Army of the Potomac. If Baker’s allegations were true, Brady, while gaining the confidence of Grant, was neck-deep in a scheme cooked up by Wall Street fat cats to gain confidential, advance information on the Union army’s efforts, giving them a trading advantage in the markets, which would go up or down depending on what the army did and whether it succeeded or not. Writing from New York, Baker told Dana: “I have just come into possession of certain facts in relation to a plan arranged and devised by a number of prominent Wall Street Brokers for procuring information from the Army of the Potomac as to the success or defeat of General Grant’s intended movements on Richmond.”2 The previous spring, Baker wrote, Brady had been with Grant’s army, taking photographs of Grant and his generals. The more than 200 photographs that Brady took on that two-week excursion to the front in June 1864 are the best documentation of that visit, including several photos of Grant, group shots of generals, the well-known stereo view of Brady sitting with Gen. Ambrose Burnside and other images with Brady in the photo. Back in Washington, Brady published a catalog of those images – a total of 231 in four different sizes, featuring 81 large photos ($1.50 each) and 84 stereo views (50 cents each). They apparently did not sell well, because extant, vintage original prints or stereo cards are exceedingly rare, indeed almost unheard of, in today’s antique were true, Brady, while gaining the was neck-deep in a scheme cooked up by Wall Street fat cats to gain confidential, advance information…”

A Brady studio portrait of Lafayette C. Baker (National Archives)

wrote, which worked. In City Point, by photographing Grant and his generals, Brady “placed himself on the most intimate terms with these officers.” 4 The scheming brokers knew this, Baker’s sources told him, and “at once selected him as a medium through which they proposed to obtain the information desired.” Baker said Brady would write a telegram to his gallery in New York, requesting one of his artists to come at once to the army to assist Brady. Although no telegrams could be sent from the army just before or during an important movement without Grant’s approval, Brady would obtain the general’s approval, but not send the telegram until “a moment when the success or defeat of our Army will materially affect the Stock and Gold market,” Baker wrote.5 Baker said he was told of the scheme by “one of the principal (supposed) actors in the scheme, and can be relied upon as being strictly true.” And since Brady had “obtained the unlimited confidence of General Grant,” Baker said he “deemed it proper to communicate these facts to the Department in order that the General may know how he is being deceived…”6 What are we to make of this explosive letter, which has remained all but hidden in the army records since the day it was filed away as an unregistered piece of correspondence? Was there any truth to it? We may never know for sure, but there is no record of any action being taken by anyone in

CivilWarNews.com connection with the allegations, and no hint of the alleged scandal in any of the books or articles about Brady. A search of National Archives records reveals no record of any telegrams sent by Brady from army headquarters in June 1864, or any time that year. There is no record of any telegrams sent by his two primary assistants, David Woodbury and Anthony Berger. Additionally, there is no record of any letter or written application from Brady to the War Department asking permission to visit the army in 1864, and no refusal letter to Brady. Both Dana and Baker wrote war memoirs, but neither mention Brady in their narratives. Dana could have provided Baker with a personal perspective of Brady’s camp activities that spring in Virginia because he was there. In fact, Dana could have been considered one of those whom Brady placed himself on intimate terms with, because Dana sat in front of Brady’s stereo camera for an individual portrait of himself in camp at Cold Harbor. This image was sold as a stereo view in the E. & H.T. Anthony & Co. War for the Union stereoscopic series. LaFayette Curry Baker, who also sat in front of Brady’s camera during the war in one of his studios, took over the U.S. Intelligence Service, forerunner of the U.S. Secret Service, in 1862 at age 36, succeeding Allen Pinkerton. A big part of his job was rounding up traitors and other criminals, and his agents would bring in several men a day, logging their names into a register of arrests, mostly for desertion, disloyalty, drunkenness, disorderly conduct, and selling liquor to soldiers.7 He was frequently accused of brutality and abuse of power in his work, and this eventually included Secretary of War Edwin M. Stanton, who had hired him. In 1864, Baker was relegated to the New York office after allegedly tapping Stanton’s telegraph lines, suspecting him of corruption. But while in New York, Baker seemed to dive into his tasks with undimmed zeal. After Lincoln’s assassination, Baker was summoned back to Washington by Stanton and placed in charge of the Booth manhunt.8 Chapter titles from Baker’s 1867 war memoir, History of the United States Secret Service, provide a telling summary of his wartime work, among them, “Treason and Traitors at the North”; “Disloyalty Among the Postmasters”; “Frauds by Government Employees and Others”; “Stealing-Smuggling Liquors Into the Army”; “Wealthy Traitors—Fruitless Schemes”; “Gigantic Vices of the National Capital”; “Counterfeiters and Counterfeiting,” and “Bounty Jumpers in Organized Bands.”9 Baker’s interest in photography seemed to be limited to how he might exploit it, and how it might be used salaciously. At the outbreak of the war, he attempted to pose as a photographer to gain access to Confederate leaders, a ruse that managed to get him arrested in Richmond. He escaped, returned to Washington, and began doing Stanton’s bidding after being appointed Provost-Marshal of the War Department in 1862.

In 1863, Baker launched a crusade against pornography. “I know of no lower grade of depravity than that of this shameless business,” he wrote. “The vile book, photograph and wood cut were scattered by sutlers, mail agents and others throughout the army. The art of photography and printing has flooded the country with these cheap and shameless appeals to the lowest and most brutal passions.” Baker said he seized $22,000 worth of the “vile goods” (about $405,000 in today’s dollars). He decided to make a bonfire of “this pile of sensual trash” and “our pure-minded President intimated that he would like to see the conflagration. It was kindled in front of the White House, and he enjoyed the sight…”10 Grant, in correspondence with his wife, Julia, took note of Brady’s presence in camp that June in Virginia. “Brady is along with the Army and is taking a great many views and will send you a copy of each,” he wrote on June 19, 1864, from City Point, just days after being photographed by Brady at Cold Harbor.11 Grant was clearly a fan of the 3-D photo. He had begun that June 19 letter to Julia by telling her he was sending her copies of the three stereo views that Timothy O’Sullivan took of Grant and his generals from the window of Massaponax Church. “To see them you will want a Sterioscope (sic),” he wrote. “Send to Covington for Buck’s, or buy one.”12 This was not the first time Grant wrote about 3-D Civil War photos while in the field. On April 3, 1863, he wrote Julia: “I sent to Buck by Capt Osband a lot of views taken at Hollysprings for him to look at through his sterioscope. I hope he has not left it behind.”13 Grant’s fascination with 3-D photos, alas, could not be shared by his wife. Julia was afflicted by strabismus, commonly referred to as crossed eyes, “which would have prevented her from experiencing the threedimensional sensation of the stereoscope,” writes Candice Shy Hooper, an independent scholar who published a book about the wives of Lincoln’s generals. Although we may never know for certain whether any insider trading scheme actually occurred that spring, we do know that Grant was not only interested in photography and stereo views, he was also a fan of stock trading. In the same June 19 letter where he mentions Brady, Grant goes on to relate to Julia: “Did I

Gen. U.S. Grant and his staff were captured by Brady outside his headquarters hut in City Point, Va. — the same building depicted on the right in the sketch on the facing page. (National Archives)

tell you that my horse rail-road stock paid 5 per cent the last quarter? This makes that $5,000 investment pay over $100.00 per month.”14 The Baker letter, although unknown within Civil War photographic history circles, has not been hidden from deepresearch scholars. Grant’s June 19 letter to Julia is published in Volume 11 of The Papers of Ulysses S. Grant. The general’s mention of “Brady” is footnoted, and at the bottom of the page, without comment, is published the entire Baker letter. This is where Candace Hooper found the fascinating account, and in 2013 she generously passed it along to us, providing yet another fascinating episode in the Civil War adventures of Mathew B. Brady. Endnotes: 1.

LaFayette C. Baker letter to Charles A. Dana, Oct.

New York City October 29, 1864

29, 1864; Letters Received, Unregistered; Records of the Headquarters of the Army, RG 108, M504, Microfilm Roll 233, National Archives and Records Administration, Washington, D.C. Ibid. “Brady & Co’s Catalogue of Photographs and Stereoscopes of Lt. Gen. Grant’s Late Campaign, June, 1864;” E 468, .7, copy on file at Prints and Photographs Reading Room, Library of Congress, Washington, D.C. Baker letter. Ibid. Ibid. “Arrests and Committals by L.C. Baker, Provost Marshal, War Department, From July 17, 1863 to October 16, 1863,” Letters Received by Secretary of War, Main Series, 18011870; March 1864-June 1864, “A and B,” Roll 250, M221, National Archives and Records Administration, Washington,

D.C. 8. John Simkin, Lafayette Baker, biography, online at http://www. spartacus. schoolnet.co.uk/ USAbaker.Laf.htm 9. LaFayette C. Baker, History of the United States Secret Service, published by L.C. Baker in Philadelphia, 1867. 10. Baker, History, page 382-383. 11. Ulysses S. Grant letter to Julia Dent Grant, June 19, 1864, Papers of Ulysses S. Grant, Volume 11 (Carbondale, Ill., Southern Illinois University Press, 1984), pp 84-85. 12. Ibid. 13. Ulysses S. Grant letter to Julia Dent Grant, April 3, 1863, Papers of Ulysses S. Grant, Volume 8 (Carbondale, Ill.; Southern Illinois University Press, 1979), page 9. 14. Grant letter, June 19, 1864. Bob Zeller is co-founder and president of The Center for Civil War Photography (www.

civilwarphotography.org), a nonprofit educational organization founded in 2001. He has published four books on the subject, including The Blue and Gray in Black and White (Praeger, 2005), the only booklength narrative history about the war’s photographers and their accomplishments.

Transcript of Brady Insider Trading Letter

Honorable C.A. Dana Asst. Secretary of War Washington, D.C. Sir: I have just come into possession of certain facts in relation to a plan arranged and devised by a number of prominent Wall Street Brokers for procuring information from the Army of the Potomac, as to the success or defeat of General Grant’s intended movements on Richmond. Some time last spring “Mr. Brady” the well known Daguerreotypist of New York, made application to Mr. Stanton for a pass to visit the Army of the Potomac for the purpose of taking views of Battle scenes, etc. The request of Mr. Brady was refused at the War Department, he Brady then applied to the President, who also refused to grant his request. A few days afterwards, Mrs. Brady wrote a letter to Mrs. General Grant, asking Mrs. Grant to use her influence with the General to procure the desired pass. This letter obtained the wished for result, viz, the procuring of the pass from General Grant. Brady went to the Army and by his gratuitous offers to take the pictures of prominent Army Officers for the Illustrated papers in this city, placed himself on the most intimate terms with these officers. The Brokers referred to, having learned Brady’s position in the Army, and knowing that he, Brady, possessed Genl. Grant’s confidence, at once selected him as a medium through which they proposed to obtain the information desired.” It was understood by the Brokers referred to that immediately preceding or during any important movement or battle, that no telegrams could be sent from the Army without General Grant’s approval or endorsement. Brady (then being with the Army of the Potomac), is to write and forward a telegram purporting to be addressed to one of his (Brady’s) Artists in this city, asking the Artist or employee to come to the Army at once to assist. This or some similarly worded telegram is to be taken by Brady to Genl. Grant for his endorsement, but to be held and forwarded at a moment when the success or defeat of our Army will materially affect the Stock and Gold market. Inasmuch as Brady has obtained the unlimited confidence of General Grant (as he alleges), I have deemed it proper to communicate these facts to the Department in order that the General may know how he is being deceived by one to whom he has granted various privileges and favors. The facts and circumstances herein stated have been obtained from one of the principal (supposed) actors in the scheme, and can be relied upon as being strictly true.

The first page of LaFayette C. Baker’s accusatory letter to Asst. Sec. of War Charles A. Dana about Brady. (National Archives)

I am sir, your obedient servant, L.C. Baker